Improve a prompt by testing one change

You ask for a summary and receive a polished paragraph that drops the only exception in the policy. Adding “be excellent” will not tell the assistant what went wrong. How can you revise the prompt in a way that teaches you something about the failure?

Before you begin: Know how to separate a task, source text, and output requirements.

Start from a failing example

Use this invented policy: “Visitors may borrow two books for 14 days. Reference books must stay in the reading room. Renewals are allowed once unless another visitor is waiting.” Ask for a summary for a first-time visitor.

Before generating anything, write the facts you need to preserve: two books, 14 days, reference-book restriction, one renewal, and the waiting-visitor exception. These form a small rubric, a set of criteria for judging the result. A rubric does not have to be a complicated scoring system; it can be five checkable questions.

A candidate summary says: “Borrow two books for two weeks and renew them once.” It preserves several facts but loses two restrictions. That tells you where to focus the next experiment.

Make the revision specific

Compare two prompts on the same source:

A: Summarize this policy in plain language.

B: Summarize this policy for a first-time visitor.
Preserve every limit, exception, and restriction.
Use up to three short sentences and only facts in the policy.

Prompt B expresses what the first output missed. It may help, but the result must be measured. Run both on several policy variants, including one without an exception and one where the exception comes at the end. Do not declare B universally better because it succeeds once.

When you compare responses, hide the prompt label if you can. Otherwise you may favor the wording you just wrote. If an assistant produces variable answers, repeated trials help you distinguish a systematic difference from a lucky output.

Resolve competing instructions

“Include every detail in exactly five words” may be impossible for a long policy. When requirements conflict, state which matters most. For example: preserve material restrictions even if the summary needs an extra sentence. A model that obeys a length limit by deleting a critical exception has followed one instruction while failing the task.

Likewise, “use only the source” and “add the latest opening hours” are inconsistent if the source has no hours and no retrieval is available. Good prompt design includes a behavior for missing information: ask, leave a clearly marked gap, or state that the answer is unavailable.

Keep an error notebook

Record the input, prompt version, output, and failed criterion. Group errors by cause. Missing exceptions suggest a different change from unsupported dates, inconsistent formatting, or answers to the wrong audience. This keeps revisions tied to evidence instead of an ever-growing collection of emphatic rules.

Some failures belong outside prompting. Exact field types can be checked with a validator. Arithmetic can be checked with code. Access to private records must be enforced by the application. A prompt can request behavior, but it is not the only control a reliable system needs.

Write a better experiment

You changed the prompt, switched the model, and shortened the source. The next summary preserves the exception. Which change helped?

Now write a third policy with a condition the prompt has not seen. Use it as a held-out check. If your revision still works, you have stronger evidence that it captures the requirement rather than fitting one sentence.

The next lesson explores examples as another way to make a requirement concrete, including how poorly chosen examples can mislead the model.

Further reading

The HELM evaluation paper motivates examining multiple dimensions of performance. Anthropic's prompting overview places prompt changes within an evaluation process.

Continue to the next lesson.