Free template
Five gates between a pilot and a production AI system
What has to exist, and who signs it, before an AI system meets real customers or acts on real records. Use it as the agenda for a go-live meeting.
The pilot that became production by not stopping
Nobody decided to scale it. The volume grew, more people used it, the business came to rely on it, and one day it was in production without any of the things a production system needs.
The gates make the decision explicit. The line is not volume. It is consequence: the moment a wrong output costs the business or a customer something, the system is in production, whether anyone decided so or not.
What has to exist, and who signs
1. The specification is signed
Signed by the owner and the engineer
The acceptable output is written so a tester could grade it. The failure modes, the limits on what it can do, and what happens when it cannot do the task are all defined.
2. The evaluation passes, reliably
Signed by the keeper of the evaluation set
Real cases, run more than once, including attempts to trick it, scored above the owner's threshold on the exact version going live.
3. The controls are in code, and tested
Signed by the engineer, and the owner for the limits
Permissions, limits and approval steps live in the system's own code. Someone has tried what each control forbids, and it was forbidden. The off switch has been tested.
4. The operations exist
Signed by whoever will be called out
End-to-end logging, a dashboard with quality and cost alerts, a cost forecast, a fallback, a runbook and an on-call rota.
5. The business can run it
Signed by the owner and the engineer together
A named owner, a named keeper of the evaluation set, a named person who reads the escalations, a review date, a switch-off condition, and support arrangements written down.
Built together, walked in order
The gates are written in order but built in parallel. The specification shapes the evaluation, the evaluation exposes gaps in the controls, and the controls produce the signals the operations watch. Built together, they are mostly passed by the time of the go-live meeting.
Before the meeting, write exists, partial or missing against each gate, with the evidence and the signatory. Make the first "missing" the next task.
The gate most often failed is the second. Teams arrive with an evaluation made of the demonstration cases and a feeling. The first proper measurement is lower than the feeling. The right response is to fix what it found, not to lower the threshold.
Using the go-live record
What may a pilot skip?
A pilot can run with a handful of friendly users, a developer's account and ten test cases, for a few weeks. It may not meet real customers, act on real records or be relied on without passing the gates.
Do we need all five for a small internal tool?
Scale each gate to the consequence. An internal drafting assistant needs a light version of each. A system that talks to customers or can act needs every one in full.
Who supports the system after go-live?
The business runs it, with a named owner and keeper. Support from the builder is fine, on terms that are written down at gate 5.
Ready for the go-live meeting?
ENGINEER chapter 22 sets out each gate in detail, with a real system walked against them. Or we can walk the gates with you.