The eval is the product

There is a particular meeting that happens about six weeks into every AI project. Someone changed a prompt, the output looks different, and nobody in the room can say whether it is better. The conversation goes in circles for forty minutes and ends with a decision made on vibes.

That meeting is a symptom. The cause is that the team built a system before they built a way to measure it.

Why the usual testing instinct fails here

Traditional software has a comforting property: the same input produces the same output, so a test is a straightforward assertion. AI systems break that assumption in two directions at once. The output varies between runs, and, more awkwardly, there is often no single correct answer to assert against.

The instinct is to give up on testing and rely on manual review. That works for about three weeks. Then the surface area grows past what anyone can hold in their head, changes start interacting, and you are shipping on hope.

What an eval actually is

An eval is three things:

• A dataset of real cases, including the ones that embarrassed you in production
• A grader that scores an output: sometimes exact match, sometimes a rubric applied by a model, sometimes a human
• A threshold that decides whether a change ships

The third one is where most teams stop short. An eval that produces a number nobody is accountable to is a dashboard, not a gate.

Start from failures, not from coverage

The temptation is to build a broad, representative test set first. It is the wrong order. Start with the twenty cases that have actually gone wrong: the support ticket that got misrouted, the invoice where the total was read off the wrong line, the answer that confidently cited a policy that does not exist.

Those cases are worth more than two hundred synthetic ones, because they encode the specific ways your domain is strange. Every production incident should end with a new row in the eval set. That habit alone puts you ahead of most teams.

Graders are where the judgement lives

For extraction tasks, grading is easy: the field is right or it is not. For anything generative you need a rubric, and writing that rubric forces a conversation that should have happened at the start: what does good actually mean here?

When we use a model as a grader, we hold it to the same standard as the system under test: we check the grader against human labels on a sample, and we re-check when we change models. A grader that has quietly drifted is worse than no grader, because it produces confident numbers that are wrong.

Put it in CI or do not bother

An eval suite that a human remembers to run is an eval suite that stops being run in month two. Ours run on every pull request that touches a prompt, a retrieval config, or a model version. Below threshold, the merge is blocked. There is a documented override for genuine emergencies and it gets used perhaps twice a year.

This is the point at which evaluation stops being a chore and starts being leverage. Once the gate exists, engineers can change things aggressively (swap a model, restructure a prompt, rewrite the chunking) because the cost of being wrong is a red build rather than a customer incident.

The uncomfortable part

Building the eval takes longer than building the first version of the feature. Every team pushes back on this, and every team that skips it spends the saved time three times over in the following quarter.

The reframe that lands: the eval is the specification. It is the only written artefact that says precisely what the system is supposed to do, in a form that can be checked. The prompt is an implementation detail. The model is a dependency you will swap. The eval is the thing you actually own.