Skip to main content

Platform

Evaluation

Classification has an easy relationship with measurement. Agent work has several acceptable answers, a large ambiguous middle, and a path that matters as much as the output.

The evaluation loopA golden dataset feeds an offline suite that scores task success, step correctness, tool-call precision, latency and cost. Passing builds go to a canary against live traffic, which emits telemetry and sampled human review. Findings return to the golden dataset. A regression gate sits in continuous integration and blocks any change that reduces measured quality.Golden datasetcurated + harvestedOffline suitetrajectory + outcomeRegression gateblocks mergeCanary in productionshadow, then sliceLive telemetry + sampled human reviewfailures become new test casesNothing here is optional. A system without a regression gate is a system nobody can safely change.
Figure 1. Failures in production become test cases. A suite that only ever grows by hand stops predicting anything.

What we measure

MeasureDefinitionWhy it is on the list
Outcome qualityRubric score against an adjudicated setThe headline claim. Meaningless without the agreement rate beside it.
Inter-rater agreementAgreement between human adjudicatorsBounds every other claim. If reviewers agree 0.8 of the time, a two-point difference is noise.
Tool-call precisionCalls made that a competent reviewer would have madeCatches the agent that queries five systems when one would do.
Tool-call recallCalls a competent reviewer would have made that were madeCatches the agent that answers confidently without checking anything.
Escalation calibrationFalse escalations and, harder, missed escalationsThe second is where real risk sits and almost nobody instruments it.
Cost per resolved taskTotal cost divided by cases actually closedPer-request cost flatters systems with a high retry or handoff rate.
Cost variance (p95/p50)Spread of cost across casesDetermines whether the system can be budgeted at all.
Replay fidelityTraces that reproduce the same terminal stateWhat a supervisory review actually depends on.

Building the dataset

A few hundred well-adjudicated cases outperform several thousand casually labelled ones. Ours contain three groups in deliberate proportion: routine cases that establish the baseline, known-hard cases collected from the people doing the work today, and adversarial cases constructed to break specific assumptions. The third group is small and does most of the work.

Every case carries documented reasoning. A test case whose purpose nobody remembers gets deleted the first time it fails, and with it goes the requirement it encoded.

Using a model as judge

It works within limits, and the limits are not optional: calibrate against human review on a held-out sample before trusting it, report the agreement rate alongside any result it produced, never use the same model as judge and subject, and keep a permanently human-reviewed subset because judge drift is silent.

What an evaluation report contains

  1. 01 · The claim, stated as something that could have been false.
  2. 02 · Dataset provenance: where cases came from, who adjudicated, what was excluded.
  3. 03 · Inter-rater agreement, before any result.
  4. 04 · Results with uncertainty, not point estimates.
  5. 05 · Judge calibration, where a model judge was used.
  6. 06 · Limitations, what this does not show, written by the person who ran it.
  7. 07 · What changes would invalidate the result.

Already have an evaluation suite?

A common first job is reading the one you already have and writing up what it can and cannot actually support. That report tends to make for unwelcome reading, but it is a lot cheaper to find out now than after go-live.

Talk to a data scientist Evaluation & assurance