Testing & QA for Non-Deterministic Agents

Your API wrappers, data transformers, and tool implementations should have deterministic unit tests - same input always produces same output. Run these in CI on every commit and catch integration bugs before they reach production.

Mock LLM responses in CI tests. Create fixtures for common agent scenarios: 'given this ticket input, the agent should call get_customer, then search_kb, then draft_response.' Test that the right tools are called in the right order with the right parameters. The model's creative output is irrelevant - the orchestration logic is what you test.

Run live LLM evals nightly or on-deploy, not in CI. Live evals cost money and have variable results, but they catch prompt regressions that mocked tests miss. Separate fast deterministic tests (CI, every commit) from slow probabilistic evals (nightly, pre-deploy).

Using an LLM to score agent outputs is useful but dangerous if it is your only quality gate. LLM judges inherit the biases of the judge model and can be gamed by agents that produce verbose, confident-sounding wrong answers.

Build rubric-based scoring with human-labeled calibration examples. Define explicit criteria: 'Accuracy: did the agent use correct information? (1-5). Completeness: did it address all parts of the request? (1-5). Tone: is it appropriate for the context? (1-5).' Calibrate the LLM judge against 50+ human-labeled examples until its scores correlate with human scores above 0.85.

Never rely solely on LLM-judge pass rates. Always include human review for a sample of production outputs (5-10% initially, decreasing as confidence grows). LLM-as-a-judge is a screening tool, not a replacement for human judgment.

Before promoting a new agent version to production, run it in shadow mode: process the same inputs as production, compare outputs, but do not show shadow results to users. Promote the new version when shadow metrics beat production baseline for at least a week.

Track shadow vs. production on: completion rate, escalation rate, average latency, cost per task, and human approval rate (for agents with human checkpoints). A new version that completes 5% more tasks but costs 3x per task is not an improvement.

Shadow mode is especially critical for prompt changes. A prompt tweak that improves response quality on 90% of tasks might break the 10% of edge cases that matter most to your best customers. Shadow mode catches these regressions before they affect real users.

Do not write test cases from imagination. Every case should come from something that actually went wrong: a customer complaint, a wrong answer someone spotted, a run that stalled. Cases invented at a desk test the paths you already thought about, which are the paths that work.

Twenty to fifty cases is enough to be useful and small enough to run often. Each one needs the input, the context available at the time, and what the correct outcome was. Store them in version control beside the code, because they are as much a part of the product as the prompts.

This set becomes the most valuable thing you own. Prompts can be rewritten and models replaced, but a library of real failures with known correct answers is earned over months and cannot be copied by anyone who has not served your customers.

Do not assert on exact strings for anything the model writes freely. Assert on the parts that must be true: the structure parses, the enum is one of the allowed values, the cited document is the right one, the tool was called with the right identifier, no forbidden claim appears.

Split every case into a deterministic half and a judged half. The deterministic half is cheap, fast, and belongs in continuous integration. The judged half, where quality is genuinely subjective, runs less often and needs a human to look at samples periodically to check the judge itself has not drifted.

Where you use a model as judge, give it a rubric with explicit criteria rather than asking whether the answer is good. Then measure the judge against human ratings on a sample. An unvalidated judge is a confident number that means nothing.

Decide the pass threshold before you need it, and write it down: no prompt or model change ships if the eval set drops below the agreed level. Deciding in the moment, with a fix you want to release, produces a threshold that moves to accommodate the release.

Compare per case rather than on the average. A mean that holds steady can hide two cases breaking and two improving, and the two that broke may be the ones a customer depends on. The per-case diff is what tells you whether to ship.

Keep the last known good result stored so a regression can be attributed to a specific change. Without that, a slow degradation across several edits becomes untraceable, and untraceable degradation is how agents quietly stop working over a quarter.

For the triage agent the split between deterministic and judged fell out cleanly. Deterministic: the category is one of the six allowed values, the cited help article exists, no reply contains a refund or credit promise, and the escalation reason is a known enum. All four are cheap and run on every change.

Judged: whether the tone suits a frustrated customer, and whether the reply actually answers the question rather than restating the policy. Those needed a rubric and a person spot-checking a sample, because a model asked whether a reply is good will say yes to almost anything grammatical.

The gate was set at no regression on the deterministic assertions and no more than one case moving in the judged set. That threshold was written down in week three, before anyone wanted to ship something urgently, which is the only time such a number gets set honestly.