What Problem Does RAG Actually Solve?

A language model only knows its training data. It also knows nothing specific to your organisation: contracts, product manuals, last week’s support tickets, the pricing sheet updated this morning. Closing that gap is the entire problem RAG exists to solve, and it is one of three ways to solve it. The other two are fine-tuning and a plain search index. Choosing between them is a business decision, not a technical one, and it should happen before anyone picks a vector database.

Fine-tuning bakes new knowledge into the model’s weights through further training. The new knowledge becomes part of the model itself. Nothing is looked up at the point of answering; the model simply behaves as though it always knew.

Retrieval-augmented generation takes a different approach. At the moment a question arrives, the system searches your own content for the passages most likely to contain the answer, places those passages in front of the model alongside the question, and asks the model to answer using that material. The knowledge lives outside the model, in whatever document store you point it at, and gets pulled in fresh on every query.

A search index skips the model altogether. The user asks a question, gets back the ranked list of documents that match, and reads the answer themselves. No synthesis, no generated prose, no model in the loop at the point the answer is produced.

Choosing between them means asking three questions:

How often does the knowledge change? Content that shifts weekly (support tickets, prices, current policy) suits RAG well, because updating an index is cheap compared with retraining a model. Content that is stable and foundational (house style, a domain’s underlying reasoning, how a classification task should behave) suits fine-tuning, since the cost of retraining is paid rarely and the knowledge does not need refreshing on any particular schedule.

Does the answer need to cite its source? RAG produces this naturally: the system already knows which passage it retrieved, so pointing to “paragraph three of the March contract” is a by-product of how it works rather than something bolted on afterwards. Fine-tuning has no equivalent. The knowledge is diffused across the model’s weights, and there is no paragraph to point to.

Does the user actually want a synthesised answer? RAG is frequently reached for in situations a well-built search box would serve better and more reliably. If someone needs the relevant clause in a policy document, returning that clause is a safer answer than asking a model to summarise it, since summarising introduces risks that simply returning the source text does not.

RAG has a significant failure mode: A model given the wrong passage will still produce a fluent, confident answer; it has no way to know the passage was wrong. Retrieval quality becomes a new measurable system property that can fail silently, in addition to whatever the model itself might get wrong.

The question worth asking before any RAG project starts is: how often the underlying knowledge changes, and whether the answer needs to show its working.

(If you are weighing RAG against fine-tuning for a specific use case, get in touch.)