Build an Eval Set That Catches Real Failures
Where cases come from, what coverage means, and the leakage that makes a suite look better than the agent it tests.
Imagined cases test what you already fixed
The instinct is to sit down and write test cases. It produces a suite that passes, because you are writing cases for the paths you already have in mind, and those are the paths you already handled.
Real cases come from things that went wrong: a customer complaint, an answer someone spotted as false, a run that stalled, an escalation that should not have happened. Each of those is a case you could not have invented, because you did not know the agent would do that.
This has a scheduling consequence. You cannot build a complete eval set before launch, so stop trying. Start with a handful drawn from your pilot, and treat every production surprise as an item to add. The suite becomes valuable at roughly the speed you meet reality.
Mine the sources you already have
Support tickets are the richest source, because a customer took the trouble to complain. Traces where the agent hit the step ceiling are second, since those are tasks it could not finish. Escalations with the reason logged are third, and the reasons cluster.
Also mine near-misses. A run a human corrected before it shipped never became a ticket, but it is exactly a case worth keeping. If your interface has an override or an edit-before-send step, those edits are a free supply of labelled failures.
Record the whole context, not just the input. A case is the input plus what the agent could see at the time: the retrieved documents, the account state, the tool responses. Replaying an input without its context tests a different situation from the one that broke.
6 more sections in this article
- Coverage means shapes, not volume
- Labelling: decide what correct means
- Leakage, and suites that flatter themselves
- Size, maintenance, and retirement
- Make it a gate, not a report
- Worked example: the ticket triage agent
This is Pro content
Get Agent Factory Pro - a one-time payment for lifetime access to full articles, complete build prompts, and everything new.
Use it
The parts of the library that put this article to work.
6 more build prompts reference this article.
Published 18 August 2026. Last reviewed 17 August 2026. We re-read this library on a schedule and date every article, so you can see for yourself how current it is.