Engineering · Advanced
How to test an AI agent: a five-stage framework
From your first twenty failure cases to a CI gate that blocks bad merges. The evaluation process we run on every production system.
Teams ship agents and then have no reliable way to tell whether a change helped. This framework closes that gap in about a week of focused work.
The order matters. Most teams start at stage two, writing graders before they have collected the failures that would tell them what to grade.
The playbook
01 Collect real failures, not synthetic ones
Pull twenty cases where the system got it wrong in front of a real user. Screenshots, logs, transcripts, whatever you have.
Twenty real failures are worth more than two hundred generated ones, because they encode the specific ways your domain is strange.
02 Write the grader
For extraction, grading is exact match on the field. For generative output, write a rubric: what must be present, what must never be, what is merely preferred.
If you use a model as grader, validate it against human labels on a sample of at least fifty. Re-validate whenever you change the grader model.
03 Set a threshold someone is accountable to
Score the current system to get a baseline. Set the pass threshold slightly above it.
A number nobody is accountable to is a dashboard, not a gate. Name the person who owns the threshold.
04 Wire it into CI
Run the suite on every pull request that touches a prompt, retrieval config, or model version.
Below threshold blocks the merge. Provide a documented override for genuine emergencies and audit its use.
05 Grow the set from production
Every production incident ends with a new case in the eval set. Make this a step in your incident template, not a good intention.
Review the suite quarterly and retire cases that no longer discriminate between good and bad systems.
Start before the gap gets expensive.
Tell us where the work is stuck. We will tell you, in plain language, what AI can and cannot fix, and what it would take to do it properly.