Confidence-based escalation in AI agents: the engineering guide

Reviewed October 2026. The design described here is the one our agent framework uses in production; the specifics (signals, thresholds, routing) vary by deployment and are set from the evaluation set, never by guesswork.

Contents: Why an agent needs to know what it does not know · What confidence is · The signals · Combining signals into a score · Setting the threshold · What happens on escalation · Hard rules that bypass the score · The feedback loop · Measuring it · Design decisions by deployment · Failure modes · Checklist

A language model will answer anything. Asked a question it cannot answer from the material it has, it will produce a fluent, plausible, wrong answer, and it will produce it with the same tone as a right one. That is the single property that makes putting an agent in front of customers, or in front of engineers making decisions, dangerous. Confidence-based escalation is the mechanism that fixes it: the agent estimates how likely its answer is to be right, acts on its own when that estimate is high, and hands the case to a person when it is not. This guide explains how that estimate is built, where the line goes, and what the people on the other side of it do.

Why an agent needs to know what it does not know

In support, the distribution of questions is long-tailed. Most are routine and the knowledge base answers them; a minority are unusual, ambiguous, emotionally charged, or about something nobody has written down. An agent that answers all of them will be right on the routine ones and wrong on the tail, and the tail is where the customers who remember are. The business value is in the routine majority; the business risk is in the tail. Escalation lets the agent take the majority and leave the tail to people, which is also where people add the most value. The same logic applies to an internal assistant advising engineers, an agent processing documents, or any system whose errors have a cost.

What confidence is

Confidence here is not the model’s own self-report. Asked “how sure are you?”, a model produces a number that correlates only loosely with being right. Confidence, as engineered, is a score composed from several independent signals that each predict correctness, calibrated against an evaluation set so that “0.8” means something like “right about 80% of the time on cases like this”. The calibration is the point: an uncalibrated score is a vibe.

The signals

The signals our framework uses, in rough order of usefulness:

Retrieval quality. Did the knowledge base contain material that answers the question? Measured by how many retrieved passages are relevant, how strongly they match, and whether they agree with each other. A question for which retrieval returns nothing relevant is a question the agent cannot answer, whatever the model says next. This is the strongest single signal.

Grounding check. Is every claim in the drafted answer supported by a retrieved passage? A second pass, often a smaller model, checks the answer against its sources and flags unsupported statements. Answers with unsupported claims are low confidence by construction.

Classifier agreement. The question is classified by intent (order status, returns, technical fault, complaint, something else) by more than one method: the model, a lightweight classifier, keyword rules. When they disagree, the question is probably ambiguous, and ambiguity lowers confidence.

Self-consistency. Ask the model the same question more than once, with variation, and compare. Consistent answers are more often right; divergent ones are a sign the model is guessing. Costs more compute; used selectively.

Policy and entity checks. Does the answer involve a refund, a date commitment, a legal statement, a price, a safety-relevant instruction? Does it mention an order, account or product the agent could not verify? These are deterministic checks and they feed both the score and the hard rules below.

Conversation signals. Repeated questions, expressions of frustration, mention of cancellation or complaint, a long thread. These predict that a person is needed regardless of whether the agent could answer.

Historical performance. On the evaluation set, how often has the agent been right on this intent, this product line, this type of question? A category where the agent’s record is weak gets a lower prior.

Combining signals into a score

The signals are combined into a single score. The simplest version is a weighted sum with weights set by fitting against the evaluation set: which signals, at which weights, best separate the cases the agent got right from the ones it got wrong. More sophisticated versions train a small model to predict correctness from the signals. Either way, the score is validated the same way: on held-out evaluation cases, does a higher score mean a higher chance of being right? If it does not, the score is not confidence and should not be used as such.

Keep the combination interpretable. When a case is escalated, the person receiving it benefits from knowing why (retrieval found nothing; the answer involved a refund; the classifiers disagreed), and when the business asks why the agent handled a case it should not have, the answer needs to be traceable.

Setting the threshold

The threshold is the score above which the agent acts alone. It is a business decision, made with the evaluation set:

1. Run the evaluation set through the agent and record each case’s score and whether the answer was acceptable.
2. For every candidate threshold, compute two numbers: the share of cases the agent would handle alone (automation rate) and the share of those it would get wrong (error rate among automated cases).
3. Plot them. The curve shows the trade: a higher threshold means fewer cases automated and fewer errors; a lower one, the reverse.
4. The business picks the point: “we will accept one unacceptable answer in two hundred automated cases” fixes the threshold, and the automation rate falls out.

Start conservative. A new deployment runs with a high threshold and a low automation rate, people see that the escalated cases are the right ones, and the threshold comes down as the evaluation set grows and the number holds. Thresholds can also differ by intent: a lower bar for order-status questions, a higher one for anything involving money.

What happens on escalation

Escalation is a product, not a dead end. A good handoff:

• Goes to the right person, routed by the intent classification: technical faults to technical support, complaints to the team empowered to resolve them.
• Carries the context: the conversation, the retrieved material, the draft answer the agent did not send, and the reason for escalation. The person starts ahead, not from zero.
• Tells the customer honestly what is happening: that a colleague will pick this up, and when. Not a pretence that the agent is “checking”.
• Is fast: the SLA for escalated cases is the one the business is judged on, because these are the cases that matter.
• Can hand back: for some deployments the person answers and the agent continues the conversation; for others the person takes it over. Decide which.

In the Agency51 deployment, low-confidence tickets go to a developer with the agent’s draft, its reasoning and the code it looked at attached; the escalation is a review with a head start, not a restart.

Hard rules that bypass the score

Some cases are escalated whatever the score, because the cost of error is not about probability:

• Anything involving money leaving the business: refunds, credits, price promises.
• Legal, regulatory or safety statements.
• Vulnerable-customer indicators.
• Explicit requests for a person.
• Topics on a maintained deny-list (an ongoing incident, a product recall, a legal dispute).
• Actions with side effects the agent has not been authorised for.

Hard rules are deterministic, cheap, and the first thing a reviewer checks. They are also where the EU AI Act’s human-oversight expectation is most visibly met; see the EU AI Act guide.

The feedback loop

Every escalated case is a labelled example: the person’s answer is the ground truth. Fed back, it does three things: grows the evaluation set, so the next threshold decision is better founded; identifies gaps in the knowledge base, because a cluster of escalations on one topic means the topic is not written down; and recalibrates the score, because the signals that predicted the agent would be wrong can be checked against whether it was. A deployment with this loop improves monthly; one without it decays as the business changes around it.

The loop needs a person whose job includes it: reviewing escalations weekly, adding to the knowledge base, and re-running the evaluation. Hours, not days, but owned.

Measuring it

The numbers a deployment reports:

Automation rate on its own is the vanity number. Automation rate at an agreed error rate is the real one.

Design decisions by deployment

The same mechanism, tuned differently:

• Customer-facing support. Conservative threshold, strong hard rules, honest handoff messaging, routing by intent.
• Internal knowledge assistant. Higher automation; escalation is “I can’t find this in our documents, here is who might know”, and the feedback loop targets the knowledge gaps.
• Document processing. Confidence per extracted field; low-confidence fields go to a review queue rather than the whole document.
• Agents that act. Confidence gates each action, not just each answer; irreversible actions require a higher score or a person, always.
• Engineering advice. Citations mandatory; escalation is to a named expert; the evaluation set is built by that expert.

Failure modes

• Uncalibrated score. The number means nothing; the threshold is theatre. Fix: validate against held-out cases before trusting it.
• Threshold set by intuition. Usually too low, because the demo looked good. Fix: the curve.
• Escalation to nowhere. A queue nobody owns. Fix: routing, SLA, ownership.
• Silent handoff. The customer is not told. Fix: honesty in the message.
• No feedback loop. The agent decays. Fix: a named owner and a weekly hour.
• Gaming by the knowledge base. Retrieval always finds something because the knowledge base contains everything, including wrong things. Fix: curate; retrieval quality is only a signal if the corpus is trustworthy.
• Over-escalation. A threshold so high the agent does nothing and the team stops trusting the escalations. Fix: the curve again, and intent-specific thresholds.

Checklist

Before an agent with escalation goes live:

• Signals are defined and each one’s predictive value is checked on the evaluation set.
• The combined score is calibrated on held-out cases.
• The threshold is chosen from the automation/error curve by the business, and recorded.
• Hard rules are written, tested and reviewed by the business owner.
• Escalation routes to named people with an SLA and carries full context.
• Handoff messaging is honest.
• The feedback loop has an owner and a cadence.
• The metrics above are reported, with automation rate always shown beside error rate.

This is the design behind our support agents and the escalation component of the framework they run on. If you have an agent without it, the evaluation set and the curve are where to start, and they are the first thing we build.