Evaluating agents when there is no single right answer
Accuracy works when there is a correct label. Most agent work has no label, several acceptable answers, and a trajectory that matters as much as the output. What we measure instead.
24 March 2026 · 3 min read · QAI Labs engineering
Classification has an easy relationship with evaluation: there is a right answer, you count how often the model finds it. Almost nothing we build in agent work is like that.
Summarise this case file. Draft a response. Decide whether this needs escalating. There are several acceptable outputs, some unacceptable ones, and a large middle region where reasonable reviewers disagree with each other. Meanwhile the path matters, an agent that reaches an acceptable answer by consulting a system it should not have touched has failed, whatever the output looks like.
Four things worth measuring
Outcome quality against an adjudicated set. Not a single label but a rubric, applied by more than one reviewer, with disagreements resolved and (critically), the inter-rater agreement reported alongside the score. If your reviewers agree only 70% of the time, your system cannot meaningfully be measured to better than that, and any claim of a two-point improvement is noise.
Trajectory correctness. Did it take a sensible path? We score tool-call precision and recall against what a competent human would have consulted: precision catches the agent that queries five systems when one would do; recall catches the agent that answers confidently without checking anything.
Cost and latency per resolved task. Per-request cost is misleading when a quarter of requests need a second attempt or a human. Divide by resolved cases and the picture changes, sometimes dramatically.
Escalation calibration. Of the cases it escalated, how many genuinely needed a human? Of the cases it resolved, how many should have been escalated? The second number is the one that matters and the one nobody instruments, because finding it requires reviewing cases the system thought were fine.
On using a model as the judge
It works, within limits, and the limits are not optional.
Calibrate against human judgement on a held-out sample before you trust it, and re-calibrate when you change the judge model. Report the agreement rate as part of any result the judge produced. Never let the judge be the same model as the system under test, the correlation in failure modes is real and it flatters. And keep a human-reviewed subset permanently, because judge drift is silent.
Used this way, a model judge lets you evaluate thousands of cases weekly instead of forty. That is a genuine capability gain. Used unexamined, it produces confident numbers that are wrong in a consistent direction.
Build the dataset before the system
The most valuable artefact on most of our engagements is not the model or the pipeline. It is a few hundred carefully adjudicated cases with documented reasoning, assembled before anyone wrote a prompt.
It is valuable because it survives everything else. Models change, frameworks change, the whole architecture may be replaced, the evaluation set still tells you whether the replacement is better. Teams that build it last end up measuring their system against whatever it happened to be good at.
Ours usually contains three things in deliberate proportion: routine cases that establish the baseline, known-hard cases collected from the humans doing the work today, and adversarial cases we constructed to break specific assumptions. The third group is small and does most of the work.
Watch it after launch
Offline evaluation tells you about the distribution you sampled. Production tells you about the distribution you have.
Sample real cases for human review continuously, at a low rate, forever. Track the escalation rate and the override rate as leading indicators, when the humans downstream start disagreeing with the system more often, something upstream has moved, and it is usually the world rather than the model. A policy document was updated. A new intake channel formats things differently. A team changed how they phrase requests.
None of that shows up in your offline suite until you put it there. Which is the point of the loop: production failures become test cases, and the suite gets slowly better at predicting the thing you actually care about.
Written by
QAI Labs engineering
Written by the engineers who did the work, reviewed before it goes out. If you want to take issue with any of it, hello@qailabs.io reaches us.