“It looked right in the demo” is not evidence, and it cannot be checked again after the next change to the model, the prompt or the data. The single artefact that separates a production AI system from a prototype is an evaluation set: a collection of real cases with agreed outcomes, run on every change, with the results reported. It is also the artefact most AI projects skip, which is one reason most pilots never reach production.
What an evaluation set is
Fifty to a few hundred real cases of the work the agent will do, each with the outcome your experts agree is correct. For a support agent: historical tickets with the reply that should have been sent, or the classification and escalation decision that should have been made. For a knowledge assistant: questions people actually ask, with the answer and the source it should come from. For a document-intelligence system: documents your lawyers have already reviewed, with the clauses and obligations they found.
The cases come from the business, not from us, and they must include the hard ones: the ambiguous ticket, the question whose honest answer is “it is not in the approved sources”, the contract clause that looks standard and is not. An evaluation set made of easy cases proves nothing.
Building one in a few hours
Pull a sample of real cases from the last year, weighted towards the categories that matter most and the ones that have gone wrong before. Sit the people who do the work with the sample and have them record the correct outcome for each, and their confidence in it. Disagreements between experts are themselves findings: they mark the cases where the business has no single right answer, and where the agent should escalate rather than decide. Three to four hours of expert time produces a usable first set; it grows from there.
What to measure
| Measure | What it tells you |
|---|---|
| Accuracy on cases answered alone | How often the agent is right when it acts without a person |
| Escalation rate | How much human review the threshold produces |
| Accuracy of escalation | Whether the cases it hands over are the ones that needed it |
| Citation rate | Whether every answer is grounded in a source |
| “Cannot find” rate | Where the sources, not the agent, have gaps |
| Cost and latency per case | What it costs to run, and whether it is fast enough |
The threshold for acting alone is then a business decision made on evidence: for any threshold, the set shows how many cases the agent would answer, how many it would get right, and how many it would escalate.
Run it on every change
Models are updated by their providers. Prompts are edited. Sources change. Each is a change that can silently make an agent worse on cases it used to get right. The evaluation set runs automatically on every one of them, and a regression blocks the change until it is understood. This is ordinary engineering discipline, continuous testing, applied to a system whose behaviour is probabilistic, and it is non-negotiable.
Let the escalations feed it
Every case a person resolves after the agent escalated it is a new evaluation case with a known-good answer, on exactly the kind of input the agent found hard. Reviewing escalations properly is therefore not overhead; it is how the agent gets better on the work it got wrong, and how the threshold earns its way down over time.
What to ask a supplier
Whether there is an evaluation set, who built it, whether it includes the hard cases, whether it runs on every change, and whether you will see the numbers. A supplier who cannot answer those is selling a demo.
The discipline is built into every framework under AI frameworks and agent build, and the escalation mechanism it supports is explained in how confidence-based escalation works.