We built our support agent for Agency51, the BigCommerce agency in our group, before we offered the framework to anyone else, and we ran it on the live helpdesk with real customers. That ordering was deliberate: a client project teaches you what the client will tolerate; your own teaches you what is true. Here are the notes, including what we would do earlier next time.
We would start the evaluation set on day one
We built the first evaluation set from historical tickets after the first prototype, which meant the prototype was judged by “it looks right”. The set should exist before the first line of code, built with the developers who used to answer the tickets, including the hard cases. Everything that followed, the threshold, the class-specific rules, the confidence that the agent was improving, depended on it. It is now the first scoping task in every build.
Indexing code by structure, not by lines
The first index chunked repositories by line count, which split functions and templates across chunks and made retrieval worse than a developer’s memory. Chunking by structure, functions, components, templates, configuration, with each chunk tagged by client and layer of the stack, was the change that made the agent genuinely code-aware. Obvious in hindsight; it cost us a fortnight.
Thresholds by class, from the start
A single confidence threshold for all tickets meant the agent was either too cautious on simple configuration questions or too bold on integration ones. Thresholds by ticket class, with some classes escalated regardless, payments, security, anything touching a live integration, is how it runs now. We would set that structure up before launch rather than discover it from the first wrong answer.
Escalation is the product
We thought of escalation as the fallback. It turned out to be the feature developers valued most: a ticket arriving with the classification, the draft, the sources and the reasoning attached is a different job from a ticket arriving cold. And every escalated case a developer resolves is a new evaluation case on exactly the input the agent found hard. We now design the handover first and the autonomous path second.
Disclosure cost nothing
We were nervous about telling customers an AI drafts first responses. Nobody minded. The combination of a fast, accurate first reply and a person one step away was received better than the slow human-only queue it replaced. Transparency, which the EU AI Act asks for anyway, turned out to be good service rather than a disclaimer.
What changed, measured
Around a hundred tickets a week triaged. First-response time from hours to minutes, because a ticket no longer waits for the right developer to be free before anyone reads it. Developers starting each ticket with the first twenty minutes done. The threshold came down over time as the evaluation set grew, class by class, on evidence.
What transfers
All of it. The framework now offered to other businesses, retrieval over code and documentation, confidence scoring, class-based escalation, an evaluation set from day one, disclosure and handover, is the one these notes produced. The case study is here, and the service under AI support and ticket-triage agents.