Iterating on Prompts
Treat prompting as a loop: test, inspect failures, refine, repeat.
You rarely get the perfect prompt on the first try. Good prompt engineering is iterative.
The loop
- Write a first prompt and run it on real inputs.
- Look at where it fails or drifts.
- Add the missing instruction, example, or constraint.
- Re-test on the same inputs to confirm you improved things.
Build a small test set
Keep a handful of tricky inputs with known good answers. Run every prompt change against them so a fix for one case does not break another.
Example
{
"v1": "Summarize the review.",
"problem": "Sometimes 5 sentences, sometimes 1; tone varies.",
"v2": "Summarize the review in exactly 2 sentences, neutral tone, no opinions."
}When to use it
- A developer runs their summarisation prompt against 20 real support tickets and refines the instruction each time the summary is too vague.
- A marketing team iterates on a tagline-generation prompt across three rounds, adding examples of rejected taglines after each run to steer output.
- A QA engineer builds a golden test set of 10 prompt/expected-output pairs and tracks which prompt version scores highest before deploying.
More examples
Compare two prompt versions
Runs two prompt versions on the same input so you can directly compare quality before committing.
from openai import OpenAI
client = OpenAI()
text = 'Photosynthesis converts sunlight, water, and CO2 into glucose and oxygen in plant cells.'
v1 = f'Summarise: {text}'
v2 = f'Summarise the following in one sentence for a non-scientist: {text}'
for label, prompt in [('v1', v1), ('v2', v2)]:
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt}]
)
print(f'{label}: {r.choices[0].message.content}')Score prompt output against golden answer
Evaluates the current prompt against a small golden set to get a numeric score you can track across iterations.
from openai import OpenAI
client = OpenAI()
golden = [
{'input': 'Capital of France?', 'expected': 'Paris'},
{'input': 'Capital of Germany?', 'expected': 'Berlin'},
{'input': 'Capital of Japan?', 'expected': 'Tokyo'},
]
correct = 0
for item in golden:
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': item['input'] + ' One word.'}]
)
answer = r.choices[0].message.content.strip()
if item['expected'].lower() in answer.lower():
correct += 1
print(f'Score: {correct}/{len(golden)}')Add failing examples to prompt
Adds a previously-failing sarcasm example to the prompt as a few-shot case, fixing the regression.
from openai import OpenAI
client = OpenAI()
# After noticing the model said 'Negative' for a sarcastic positive, add it as an example
prompt = """Classify sentiment as POSITIVE, NEGATIVE, or NEUTRAL.
Text: What a great way to ruin my morning! (sarcasm)
Sentiment: NEGATIVE
Text: {text}
Sentiment:"""
test_text = 'Oh sure, because waking up at 5am is totally my idea of fun.'
r = client.chat.completions.create(
model='gpt-4o-mini',
messages=[{'role': 'user', 'content': prompt.format(text=test_text)}]
)
print(r.choices[0].message.content)
Discussion