Write the eval before you write the prompt

Here is the loop most AI features get stuck in. Someone writes a prompt. It mostly works. A complaint arrives. They edit the prompt. It fixes that complaint and quietly breaks two things nobody noticed. Repeat for four months.

The missing piece is an eval: a fixed set of inputs with known-good outputs, and a way to score a run against them. It is the difference between changing a prompt and improving it.

Nobody skips this because evals seem like a bad idea. They skip it because building one means saying out loud what a good answer is, and on most features nobody has decided. A prompt stays vague enough to feel agreed; an eval cannot.

Twenty examples is enough to start

People imagine evals as large research undertakings. Start with twenty rows in a spreadsheet. Input, expected output, and a note on why that output is right. Twenty is enough to catch a regression, and small enough that you will actually build it today.

Where to get the twenty: your complaints. Every time someone says the feature got something wrong, that is a row. Real failures are worth ten invented ones, because they cluster around the things your inputs actually do.

The third column is the one people leave blank and the one doing the work. Writing down why an output is right is where you find that two people disagree. A reason you cannot put in a sentence is a preference, and preferences cannot be scored.

Do not make all twenty failures. A set built only from disasters says nothing about the ordinary case, and a prompt tuned on hard rows gets worse at easy ones. Twelve ordinary inputs, five known failures, three you are honestly unsure about.

Deciding what counts as right

This is the hard part, and it is hard because it forces a product decision you have been avoiding. Three grading approaches, cheapest first:

Exact match works when the output is a label, a number, or a code. Classification, extraction, routing. Cheap, unambiguous, run it on every commit.

Contains-the-facts works for summaries and answers. You list the three facts a correct answer must include and check for each. Less brittle than exact match, still mechanical.

Judge — a second model scoring the first against a rubric — works for tone, helpfulness, and writing quality. Use it last. It costs money per run and it needs its own validation: spot-check the judge against your own ratings on ten rows before you trust it.

Pick the cheapest one that captures what you care about. Most teams jump to a judge for things exact match would handle.

You need not pick one method for the whole feature. Score field by field, and a failure tells you which part broke rather than only that something did.

That is the opening. The rest of the lesson — the remaining sections, the exercise and the check-your-understanding questions — is behind the gate below.