Running an AI agent on our own agency's helpdesk before offering it to anyone else taught us things a client project would not have. The decisions we would repeat, and the ones we would make earlier.
Support queues, content at scale, reporting, SEO and now AI search: an agency's time goes on workflows with a codebase or a data source behind them, which makes agencies unusually well placed for tailored AI. What we did with our own.
"Human in the loop" is a slogan until someone decides where the loop closes. The real design question is the threshold: how sure must the system be before it acts alone, and who decides that, on what evidence.
An agent that cannot be measured is not ready for production. What an evaluation set is, how to build one with your own experts in a few hours, what to measure, and why it has to run on every change.
The engineering story behind our own helpdesk agent: why a generic chatbot could not do the job, how code and documentation are indexed so retrieval is grounded in the actual build, and the decisions that made it safe to put in front of customers.
Retrieval-augmented generation answers from passages that look like the question. GraphRAG adds the relationships between things. Which one a knowledge assistant needs depends on the questions people actually ask.
The words are used interchangeably, which is why AI conversations go in circles. One scenario shown at four levels makes the shift from AI that assists people to AI that owns outcomes concrete, and tells you which level each use case should reach.
The mechanism that makes an AI agent safe to put in front of customers: how confidence is scored, where the threshold goes, what gets handed to a person, and how the evaluation set keeps it honest.
Off-the-shelf AI products force your business into their shape. A framework is a proven architecture with the hard parts solved, tailored to your systems in weeks. Here is the difference, and when each is right.