Reviewed October 2026. Written from the engineering side: what we have seen go wrong in pilots we inherited, and what we now put in scope so it does not.
Contents: The shape of the problem · Reason 1: no evaluation set · Reason 2: no data foundation · Reason 3: no owner on the business side · Reason 4: no escalation design · Reason 5: no production path · The three quieter reasons · What to put in scope · The pilot-to-production checklist · A pilot that worked · If you have a stalled pilot now
Every survey of enterprise AI reports the same thing in different numbers: most pilots never become running systems. The figures vary with who is asking and how they define a pilot, but the gap is large and consistent, and it is where most AI budget goes to die. The demo worked. The steering group was impressed. Then nothing. Having inherited a number of these, we think the causes are specific and mostly decided before the pilot starts, in how it is scoped.
The shape of the problem
A pilot is built to answer “can this work?” and it answers yes. Production asks a different question: “does this work reliably enough, on our real data, inside our real systems, with our real people, that we can depend on it?” A pilot scoped only for the first question has no evidence for the second, and so the decision to proceed has nothing to rest on. The steering group, reasonably, asks for more evidence, and the pilot becomes a second pilot, and then a programme of pilots. The failure is not technical. It is that the pilot was never designed to produce the evidence production needs.
Reason 1: no evaluation set
The most common cause, and the least visible. The pilot was judged by looking at outputs and nodding. Nobody built the set of a few hundred real inputs with known correct answers that lets you say “it is right 94% of the time, and here are the 6%”. Without it:
- nobody can say whether the system is good enough, so the decision defaults to no;
- nobody can tell whether a change made it better or worse, so iteration stops;
- nobody can show the sceptic in the room the number they are asking for.
The evaluation set is the single most valuable artefact a pilot produces, more valuable than the pilot itself, because it is what every future version is measured against. It takes a domain expert a few days to build and it belongs in scope on day one. We wrote about building one in evaluating an AI agent before you trust it.
Reason 2: no data foundation
The pilot ran on an export. Someone pulled a CSV, or a folder of documents, or a month of tickets, and the pilot worked on that. Production needs the live data: an integration, permissions, refresh, the edge cases the export did not include. The gap between “worked on the export” and “works on the system” is often larger than the pilot itself, and it was not scoped, so it is a surprise, and surprises stall programmes.
The fix is to decide the data position before the pilot (see data readiness): is the data reachable now, reachable with work, or not reachable? A pilot on data that is not reachable is a demo, and should be called one. Where it is reachable with work, that work is in the pilot’s scope or the production path’s, explicitly.
Reason 3: no owner on the business side
The pilot was the AI team’s, or the consultancy’s, or IT’s. The function it serves turned up to demos. When the pilot ended, there was nobody in the function whose job it was to take it on, train the team, handle the exceptions and ask for the next version. Systems without an owner do not get adopted, and systems that are not adopted do not reach production whatever their accuracy.
The fix is a named business owner, in the function, from the start, with their time freed and their name on the success criteria. If no one in the function will own it, that is a finding: the function does not want it, and the pilot should not run.
Reason 4: no escalation design
The pilot handled the easy cases and someone said “we’ll deal with the hard ones later”. Production is the hard ones. A customer-facing agent with no escalation path cannot be put in front of customers; an internal assistant with no “I don’t know” is distrusted after its first confident error. Escalation is not a feature added at the end; it is the architecture: how confidence is scored, where the threshold sits, what gets handed to a person and how, and how the handed-over cases flow back into the evaluation set. We wrote it up in how confidence-based escalation works.
A pilot without escalation design produces a system that can only ever be a pilot, because the risk of putting it in production is unbounded.
Reason 5: no production path
Nobody wrote down what production would take: hosting, integration, security review, monitoring, support, cost per month, the team that runs it. So when the pilot succeeded, the next step was a blank page, and blank pages take quarters. Worse, the pilot was often built in a way that cannot go to production (a notebook, a personal API key, a prototype framework) so “productionise” means “rebuild”.
The fix is to scope the production path alongside the pilot, even if it is not yet priced: the target architecture, the integrations, the operating model. And to build the pilot on the framework it will run on in production, so that the step from pilot to production is configuration and hardening rather than a rewrite. That is the main reason we build on a framework we have already run in production.
The three quieter reasons
Success criteria nobody agreed. “Let’s see what it can do” is not a criterion. A pilot needs a number, agreed beforehand, that means proceed: accuracy on the evaluation set, hours saved per week measured, resolution rate, whatever fits. Without it the decision becomes a matter of opinion, and opinions are divided.
The wrong first opportunity. Pilots chosen for impressiveness rather than for readiness: high value, hard data, high stakes. The best first pilot is in the readiest position on the pyramid, not the most exciting; it exists to prove the method and build the muscle, and a second, bigger pilot follows faster because of it.
Too many things changing at once. A pilot that introduces a new platform, a new process and a new team simultaneously has three ways to fail and no way to tell which one did. Change the minimum.
What to put in scope
The scoping document for a pilot that is designed to reach production has these sections, each with a name against it:
- The opportunity and its pyramid layer, from discovery, with the data position confirmed.
- Success criteria: the number that means proceed, and how it will be measured.
- The evaluation set: who builds it, how many cases, when it is ready (before the build starts).
- Data: what is reachable now, what needs an extraction path, and who does that work.
- Escalation and oversight: how confidence is scored, where the threshold starts, who handles handoffs.
- The business owner and the hours they and their team commit.
- The production path: target architecture, integrations, security, hosting, monitoring, operating model, and an estimate of run cost; not yet priced in detail, but written.
- The platform: the framework the pilot is built on, which is the one production will use.
- Governance: risk category, data protection, transparency; four fields.
- The decision gate: the date, the people and the question.
A pilot scoped this way takes a little longer to start and a lot less time to finish, because the finish is defined.
The pilot-to-production checklist
At the decision gate, the pilot proceeds if:
- It meets the success criterion on the evaluation set, and the failures are understood.
- It runs on live data through the integration it will use in production, or the extraction path is built and tested.
- The business owner wants it and their team has used it.
- Escalation works: low-confidence cases reach a person, and the person’s answers flow back.
- The production architecture is written and reviewed by security.
- Run cost is estimated and someone has budget for it.
- The operating model names who monitors, who fixes, who improves.
- The governance fields are complete.
Eight ticks and it is a production decision with evidence. Fewer, and you know exactly what the remaining work is, which is itself a better position than most pilots end in.
A pilot that worked
The Agency51 support agent was scoped with an evaluation harness built from historical tickets, an owner at the agency, confidence-based escalation from the first version with escalated tickets going to a developer with the draft attached, and the same framework in pilot and production. The step from pilot to production was hardening and integration, not a rewrite, and the evaluation set is what the team still measures each change against. The decision at the gate was short, because the evidence was there.
If you have a stalled pilot now
Most stalled pilots can be rescued, and the rescue is cheaper than a restart. Run the checklist above against it. Usually two or three items are missing, and they are usually the evaluation set, the data path and the production path. Build the evaluation set first; it tells you whether the pilot is actually good, which decides whether the rest is worth doing. Then the data path. Then write the production path and take it back to the decision gate with the evidence.
If the pilot is not good, the evaluation set tells you that too, and a clean stop with a clear reason is a better outcome than a third pilot. Our agent development service covers both the scoping of new pilots and the recovery of stalled ones; the first conversation is a review of what you have against this list.