You created an agent who answers clients’ questions using the company’s knowledge base. At first it looks perfect. Then a client asks for something about returns and the agent quotes an old policy. He didn’t invent everything. He recovered the wrong source.
This is the paradox of the Retrieval-Augmented Generation (RAG): it reduces hallucinations, but does not eliminate the problem of the quality of sources.
Retrieval is a decision
When an agent recovers a source, he is already making a decision. He decides which documents matter and which ones do not. He decides whether an updated policy beats an old manual. He decides whether a short FAQ is more useful than a long procedure.
If you don’t measure this step, just measure the final output. It’s too late.
You have to draw at least:
- Agent’s query;, Candidate documents;, Selected document;, Version of source;, Retrieving confidence;, Final response;, Human or user feedback.
This telemetry allows you to understand if the problem is in the template, documents or structure of the index.
Knowledge base is not neutral
An AG agent reflects the order of your archive. If the documentation is duplicated, obsolete or badly written, the agent amplifies that mess.
Before asking AI to “reply better,” it is better to clean:
- delete outdated documents; 2. assign an owner to each source; 3. add dates and versions; 4. separate official policies from internal notes; 5. create examples of real questions.
The RAG is not just semantic technology. It is governance of knowledge.
Measure utility, not only correctness
An answer can be correct but not useful. For example: the agent quotes the right policy but does not explAIn the next step. Or uses a valid source but too technical for a customer.
Different metrics are needed for this:
- accuracy of the source;, completeness of the response;, time saved;, rate of escalation;, reopening of the ticket;, satisfaction of the customer or team.
Growth engineering brings here a precious discipline: every improvement must be linked to an observable behavior.
How to apply it without complicated work
To make RAG and Growth Engineering practical, do not start with the newest tool. Start with the point where the team is wasting time, discuss without data or make decisions with incomplete information. You can see them immediately if the theme has operational value or if it’s just a nice slide idea.
The rule is simple: an agent should not be treated as a brilliant chat. It must have clear inputs, limited tools, controlled memory and an explicit rule to pass the decision on to a person when the risk rises.
A useful sequence is this:
- Define which data the agent can read and which one should not touch; 2. Write the expected result in verifiable form, not as a general intention; 3. Decide when it needs human revision before sending or saving output; 4. Measure time saved, avoided errors and cases where the agent stops.
What to measure to see if it works
The right question is not “have we used AI?” or “have we added a new dashboard?” The right question is: what decision has become faster, clearer or safer? If it does not change a decision, the project risks remaining technical decoration.
It measures at least three levels: the spared operating time, the quality of the result and the confidence of the team in the process. Time alone can deceive: a faster but less controllable flow is not an improvement. Quality alone can deceive: a perfect system but too slow does not really enter everyday work.
The human point
An agent that cites reliable sources is not only more accurate. It is more respectful. It does not ask the user to trust a synthetic voice. It shows him that behind the answer there is a verifiable basis.
The real question isn’t, “Does our agent use RAG?” The question is, “When he recovers a source, can we trust why he chose it?”
Connection with the ginnytech path
If you want to turn this reasoning into practical competence, link this article to the path Agentic AI Data Works. The goal is not to learn new terms, but to build a way of working in which data, models and people cooperate without losing control.