Agency51 is the BigCommerce agency in our group, and its helpdesk handles around a hundred tickets a week from dozens of ecommerce brands on builds that are, by design, bespoke: custom themes, custom apps, integrations with ERPs, couriers and payment providers. The hard part of most tickets was never writing the reply. It was working out what the customer was asking and where in their build the answer lived, which meant waiting for the one developer who knew. We built an agent to do that first twenty minutes. This is how.

Why a generic chatbot could not do it

A helpdesk chatbot trained on BigCommerce’s public documentation can answer questions about BigCommerce. It cannot answer a question about this client’s checkout customisation, because the answer is in a theme file it has never seen, and it cannot tell a configuration question from a bug in a custom app, because it does not know the app exists. The knowledge that matters is the agency’s own: the repositories, the deployment notes, the past tickets. So the agent had to read them.

Indexing code and documentation together

Each client’s repository and documentation are parsed and indexed so that retrieval is grounded in the actual build. Code is chunked by structure, functions, components, templates, configuration, rather than by line count, and tagged with the client and the part of the stack it belongs to. Documentation, deployment notes and resolved tickets are indexed alongside, so a question about “the delivery options not showing” can retrieve the shipping configuration, the theme template that renders it, and the last time someone fixed something similar.

Retrieval is filtered by client before anything else happens. The agent for one brand never sees another’s code.

Classify, retrieve, draft, score, act

When a ticket arrives the agent classifies it, configuration question, bug, feature request, billing, or cannot place, because classification decides which knowledge to retrieve and which escalation rules apply. It retrieves the relevant code and documentation, reasons over it to work out what the customer is seeing and why, and drafts a reply in the agency’s tone with the specific steps for that build, citing what it used.

Then it scores its own confidence, not by asking the model how sure it is, which is unreliable, but from measurable signals: retrieval quality, agreement between the classifier and the retrieved sources, and self-checks of the draft against those sources. Above the threshold, the reply goes to the customer and the ticket is tagged for light review. Below it, a developer gets the ticket with the draft, the reasoning, the sources and the score attached, and starts from a head start rather than from nothing.

The decisions that made it safe

An evaluation set before build. Real historical tickets with known-good answers, agreed with the developers who used to answer them, run on every change to the agent so nothing regresses silently.

A conservative threshold that earned its way down. At launch the agent escalated most things. As the evaluation set grew and accuracy held, the threshold came down, class by class; integration-related tickets kept a higher bar than configuration questions.

Escalation as the feature, not the failure. Every escalated case a developer resolves goes back into the evaluation set, so the agent improves on exactly the tickets it got wrong.

Disclosure. Customers are told an AI assistant drafts first responses and that a person is always reachable.

What changed

First-response time went from hours to minutes, because a ticket no longer waits for the right developer to be free before anyone has read it. Developers see tickets with the analysis already attached and spend their time on the cases that need them. The agent does not replace them; it gives them the first twenty minutes of every ticket back.

What transfers

Nothing here is specific to BigCommerce or to agencies. Any business that supports its own software has the same problem: the answers are in a codebase and a documentation store a generic chatbot has never read. The same framework, retrieval over code and docs, confidence scoring, escalation and an evaluation set, is what we now offer as the AI support agent service, and the full write-up is in the Agency51 case study.