When you build an AI application, the question usually arrives too early.
Should we use RAG, fine-tune a model, or build an agent?
At that point there may be no test cases. Nobody has seen where the current prompt goes wrong, or agreed on what a good answer looks like. So teams pick an architecture because it is trending, or because someone already knows it.
Start with a less interesting question:
What is failing now?
To answer this, run twenty to fifty real-world scenarios through your system and carefully examine where things break down. Categorise the failures:
-
Missing facts: The model relies on outdated or incomplete information and can’t point to a source for its answer. That points to retrieval: fetch the evidence at request time.
-
Consistent mistakes on stable tasks: The model makes the same mistake on a well-defined task, even after you clarify the instructions. Fine-tuning may be worth it.
-
Dynamic decision-making required: When the right action depends on information found along the way, and each step may change as new data appears, that is where an agent fits.
For example, a support assistant might look up an order, ask for more details, or hand over to a human, depending on what the customer just said.
RAG, fine-tuning, and agents are not levels of maturity. One does not graduate into the next. Each fixes a different kind of problem, and each fails in its own way.
- Missing evidenceRetrieval / RAGFetch current, permitted sources.Watch for irrelevant or misread evidence.
- Persistent task errorsFine-tuningTrain a stable, measurable behaviour.Try instructions and examples first; verify with held-out evals.
- Next step is unknownAgentChoose actions as facts emerge.Bound permissions, time, and cost.
Combine techniques when each solves an observed failure. A fixed workflow may be enough.
Use retrieval when the model is missing evidence
Suppose an e-commerce assistant tells a customer they can return a product within 30 days, but the company recently updated its policy to only 7 days for certain items. Because return policies can change frequently and may vary by product or promotion, it’s important to retrieve the latest policy at the time of the request rather than relying on outdated information.
That is not a reason to train the policy into the model. Retraining every time a policy changes is slow and expensive, and facts stored in weights are hard to inspect or correct. The model may remember an old policy without giving anyone a useful way to find out why.
RAG is useful when answers need to reference information that changes frequently or depends on the specifics of a request, and that data can be retrieved under proper access controls. It puts the relevant source material into the model’s context before it answers. When done well, this approach allows users to trace the answer back to its original source.
Retrieving a document does not make the answer correct.
The retriever might surface information from the wrong region. Important exceptions can get lost when paragraphs are broken into chunks. Different documents may contradict each other. The model could reference the correct section but still misinterpret its meaning. Even with a confirmed citation, you only know what was retrieved, not whether the final answer is actually supported by that evidence.
Before reaching for RAG, consider these three questions:
- Is the needed information actually present in the source? Retrieval cannot surface details that don’t exist in any available document.
- Should the answer come from a document, or should it come directly from a system of record? Retrieve a return policy, but query the commerce system for an order status.
- Can you retrieve the right document? A valid policy for the wrong seller, product, region, or date is still the wrong evidence.
Access control belongs in the retrieval path too. The question is not only whether a document is relevant. It is whether this user is allowed to retrieve it now. The corpus is also untrusted input; a document can contain instructions intended to steer the model. Those are system concerns, not chunking details.
Fine-tune when the model keeps making the same mistakes
Imagine the correct policy is already present in context, but the assistant repeatedly categorises urgent safety issues as ordinary complaints. Errors like these often arise from a business-specific taxonomy that is clear to experts but not yet internalised by the model.
This is where fine-tuning might become necessary. But it should not be the first reaction.
Start by clarifying the instructions, adding tough real-world examples, and tightening the output format. Use code to validate the structure and allowed values, and hold back some cases to check that the fix works on examples it hasn’t seen.
Most importantly, start building an evaluation set (evals) early. Include today’s failures and the cases you expect as the product changes. Run every change against it. That is how you catch regressions, and it builds the habit of testing on real examples.
Fine-tuning makes sense if, after all of this, you still see a measurable, persistent behaviour that won’t budge with prompt engineering, or if you need a smaller model to reliably perform a narrowly scoped task for efficiency.
This only works when the task is stable and measurable: enough representative examples, evals with cases the model hasn’t seen, and requirements that won’t change next month.
Training changes the model’s tendencies. It does not turn changing facts into safe memory, remove the need for instructions, or guarantee a schema.
It can also make a failure more consistent. If the training data carries an old policy or a weak label, the model may reproduce it more consistently and in your preferred tone. Running evals before and after each training run is how you catch this.
Another practical question: who owns the next training run? Someone has to prepare the data, run the training, and repeat it as products, language, and edge cases change. Fine-tuning is not set-and-forget.
Use an agent when the next step can’t be planned in advance
If your process is something like “look up the order, check the delivery date, draft a reply”, simply document those steps and implement them as a fixed workflow. The model’s role can be limited to composing the reply—it doesn’t need to determine or design the process itself.
An agent becomes necessary when each next step truly depends on what has just been discovered, and the process cannot be fully predetermined even though actions and constraints are clearly defined.
The agent might review an account, decide which source to check, detect inconsistencies, gather additional facts, and determine for itself when it has sufficient information. Here the model decides the order of work, not only the wording of the reply.
That flexibility also introduces common failure patterns:
- Loops can persist longer than intended.
- Tools may deliver incomplete results.
- Actions can get unnecessarily repeated.
- A believable but incorrect intermediate finding can send the process off course.
- Costs and delays can compound over the entire workflow, not just a single step.
- When something goes wrong, the only record may be a long trace that doesn’t show why a step was taken.
More prompt text won’t fix this. Put the same controls around it that you would put around any service:
- Limiting permissions.
- Setting explicit budgets.
- Enforcing timeouts.
- Making writes reversible where you can.
- Providing useful tracing.
- Defining clear stopping conditions.
Actions such as payments, external messages, or account changes should need approval before they run. Once an irreversible action has happened, human review is only an audit.
Most products need more than one
This is not a decision tree in which only one answer is allowed.
A support platform could fetch the latest policy, rely on a fine-tuned classifier to handle a well-defined internal taxonomy, and employ an agent only for those rare, unpredictable cases that cannot be addressed by a fixed process. Most requests may still follow a straightforward workflow. Any added complexity should directly resolve a specific failure uncovered by the simpler system.
Not every failure requires one of these three solutions. Use a validator when output must strictly adhere to a specification. Prohibit actions directly in code where needed. Correct issues at the source if the underlying data is inaccurate. And if there’s no consensus on the expected answer, further model development will just automate disagreement rather than resolve it.
Our approach is straightforward: we begin with the simplest system we can test—plain code, a prompt, or perhaps a single read-only tool. We gather failures from real cases and group them without immediately settling on a preferred technology. Then, we improve the part of the system responsible for the group of failures we cannot accept.
When the model is missing facts, add retrieval. If the model keeps making the same mistake on a well-defined task, and better prompts and examples don’t fix it, fine-tuning may be appropriate. When the workflow must adapt dynamically—deciding next steps based on evolving intermediate results—reach for an agent.
Practical constraints like cost, latency, privacy, or reversibility can rule out any of these options.
Pick the technique that fixes the failure you actually have. Then make sure you can see the next one.