Free template
Test your AI on twenty real failures before you scale it
A spreadsheet for your first evaluation set: real cases written as tests, each with what a correct answer looks like, and a threshold the business owner sets before testing starts.
Passing once is not passing every time
A pilot handles the demonstration cases, so it goes live. Months later the inputs have changed, the model supplier has shipped an update, and customers find the problem first.
An evaluation set is a list of test cases: a request, the context the system would have, and what a good answer looks like. You run the system against it and count. It is the AI version of the acceptance tests you would expect of any other system.
The number that matters is how often it gets the case right every time. An agent that succeeds three times in four on one run will succeed on all of three runs only about four times in ten.
Start with twenty, from real failures
- Ten cases from the specification. Each sentence describing a good output becomes a check. Each failure mode becomes a case.
- Ten from real failures. The inputs the pilot got wrong, graded by someone who knows the task.
- Write each so two experts would agree on the verdict. If they would not, the specification is not finished yet.
- Mix the cases. Easy ones to catch regressions, hard ones, edge cases (empty, another language, out of scope, an attempt to trick it), and the facts that must survive exactly, such as an order number.
- Run each case three times. Report how often it passed every time, not just once.
- Let the owner set the threshold before the results are in.
Then keep adding. Every escalation a person overturned, every correction and every complaint is a candidate case. A set the system passes completely has stopped measuring anything.
Three ways to mark a case
Use the cheapest grader that is accurate enough for each check, and compare it with a person on a sample.
Code
Exact and cheap. A field matches, a number is right, a forbidden word is absent, a citation resolves.
A second model
For tone, helpfulness or summary quality, scored against a rubric. Check it against a person regularly.
A person
The most accurate and the most expensive. Used to build the set and keep the other two honest.
Before every change, and every week
- Before any change ships: a prompt edit, a retrieval change, a new tool, a threshold.
- Every week against the live model, because providers change behaviour. A score that moves when your system did not is the signal.
- On a sample of real production outputs, continuously, as the source of new cases.
Measure more than accuracy: consistency across runs, whether it declines what it should, cost and speed per case, and exact checks on the facts that must not change.
Name a keeper for the set. The best keepers are often in the team that sees the failures first, not in engineering. See evaluating an AI agent before you trust it.
Using the spreadsheet
How many cases do we need?
Twenty to fifty well-chosen cases from real failures is a good start. Fifty chosen with care tell you more than five hundred picked at random.
Who should own the threshold?
The business owner of the system, set before testing. The threshold is the business's tolerance for error, written as a number.
What does it cost?
Running a hundred cases three times costs a few pounds and takes minutes. The real cost is an expert's time to write the answers: days to start, hours a month after.
Measure it before you scale it
ENGINEER chapter 18 covers graders, transcripts and a real evaluation run in full. Or we can build the evaluation with you.