Agency51 is a York-based ecommerce agency and BigCommerce Preferred Partner, and part of the PureNet Group alongside Enceladus. We say that first because it matters: this agent runs on our own agency’s helpdesk, with our own clients, and the numbers below are ours. It is the proof behind the AI frameworks we offer to other businesses.
The Challenge
Agency51 supports dozens of ecommerce brands on BigCommerce builds that are, by design, bespoke: custom themes, custom apps, integrations with ERPs, couriers and payment providers. Support tickets arrive through the helpdesk at a rate of roughly a hundred a week, and the hard part of most of them is not writing the reply; it is working out what the customer is actually asking and where in their build the answer lives.
That knowledge sat with a small number of developers. First responses took hours because a ticket had to wait for the right person to read it, open the repository, and work out whether this was a configuration question, a bug, a feature request or a misunderstanding. Simple questions queued behind complex ones. The developers’ time went on triage rather than fixing.
A generic helpdesk chatbot would not have helped. It could answer questions about BigCommerce in general; it could not answer questions about this client’s checkout customisation, because the answer was in code it had never seen.
Our Approach
We built an agent from the support-and-triage framework rather than a product: a proven architecture, tailored to Agency51’s systems in weeks.
The agent has access to each client’s repository and documentation, indexed so that retrieval is grounded in the actual build. When a ticket arrives it:
- Classifies the request: configuration question, bug report, feature request, billing, or something it cannot place.
- Retrieves the relevant code, configuration and documentation for that client, and reasons over it to work out what the customer is seeing and why.
- Drafts a reply in the agency’s tone, with the specific steps or explanation for that client’s build.
- Scores its confidence, from retrieval quality, classifier agreement and self-checks against the draft.
- Acts on the score. Above the threshold, the reply goes to the customer and the ticket is tagged for a light-touch review. Below it, the ticket is escalated to a developer with the agent’s draft, its reasoning and the code it looked at attached, so the person starts from a head start rather than from nothing.
The threshold was set conservatively at launch and tuned against an evaluation set of real historical tickets with known-good answers, which is run on every change to the agent.
What We Delivered
- A production agent built on Claude within our support-agent framework, integrated with the helpdesk, the client repositories and the documentation store.
- Confidence-based escalation with a tunable threshold and an audit trail for every decision.
- An evaluation harness of historical tickets, run on every change, with resolution accuracy and escalation rate reported weekly.
- Human-in-the-loop review on every escalated case, which also feeds the evaluation set.
- Transparency to customers that an AI assistant drafts first responses, with a person always reachable.
The Outcome
First-response time moved from hours to minutes, because a ticket no longer waits for the right developer to be free before anyone reads it. Developers now see tickets with the agent’s analysis already attached, and spend their time on the cases that need them.
The threshold started high and came down as the evaluation set grew; every escalated case a developer resolves goes back into that set, so the agent improves on exactly the tickets it got wrong.
The agent does not replace the developers; it gives them the first twenty minutes of every ticket back.
Why It Matters Beyond Agency51
The framework is not specific to BigCommerce or to agencies. Any business that supports its own software, a SaaS product, an internal platform, a bespoke integration, has the same problem: the answers are in a codebase and a documentation store that a generic chatbot has never read. The same architecture, retrieval over code and docs, confidence scoring, escalation to people and an evaluation set, is what we now offer as the AI support agent framework.