Every responsible-AI framework says “human in the loop”. Almost none say where. A person can review every output, which is safe and does not scale; a person can review none, which is fast and reckless; or a person can review the cases the system is not sure about, which is the only version that works in production. That third version depends on one number, the confidence threshold, and deciding it is the most important governance decision in any agent build.
Three places a person can sit
Before the action, on everything. The system drafts; a person approves each one. This is the copilot level, and it is the right end state for work where judgement is the point: editorial decisions, safety assessments, anything where the person’s accountability is the product. It does not reduce the queue; it makes each item in the queue faster.
Before the action, on the uncertain cases. The system acts alone above a confidence threshold and hands over below it, with its draft, its reasoning and its sources attached. This is where production agents live, and where the human effort goes exactly where the uncertainty is.
After the action, by sampling and audit. The system acts; people review a sample and investigate anomalies. Appropriate for low-stakes, high-volume work with a reliable audit trail, and only once the second mode has earned the trust.
Most systems move from the first to the second over time, and some to the third. The movement is governed by evidence.
The threshold is a business decision
Confidence is scored from measurable signals, retrieval quality, classifier agreement, self-checks against sources, and compared with a threshold. Where the threshold sits is not a technical question. Run the agent across an evaluation set of real cases and, for any threshold, the set shows how many cases the agent would answer alone, how many of those would be right, and how many it would escalate. The business then chooses the point on that curve: how much human review it is prepared to do, against how much risk of a wrong answer reaching a customer, a supplier or a regulator.
That choice should be made by the people accountable for the outcome, with the curve in front of them, and recorded. It is the human-oversight field on the use case’s business case, made concrete.
Different thresholds for different classes
One threshold for everything is wrong, because the cost of an error differs by case. A configuration question answered slightly imprecisely costs a follow-up message; a payment or security question answered wrongly costs far more. So thresholds are set by class, and some classes are escalated regardless of confidence. At Agency51 the integration-related tickets kept a higher bar than configuration questions throughout.
The threshold moves
It starts conservative, so the agent escalates most things and earns trust. Every escalated case a person resolves becomes a new evaluation case; the set grows on exactly the inputs the agent found hard; accuracy on those improves; and the threshold comes down, class by class, on evidence. It can also go up: a model update, a change in the sources or a new kind of ticket can lower accuracy, and the evaluation run that catches it should raise the threshold until the cause is fixed.
What the EU AI Act asks
For high-risk systems the Act requires human oversight that is effective: people who understand the system, can interpret its output, and can intervene or stop it. The threshold model satisfies that in a way that “a person checks everything” does not, because it is measured, documented and reviewed. For limited-risk systems the duty is transparency, which the handover provides: the customer knows it is AI and a person is one step away.
The mechanism is explained in how confidence-based escalation works, and the build discipline around it under AI agent development.