RAG or CAG: the operational choice before the architecture

When a team wants to build an AI assistant on top of its own documents, the first proposal is almost always the same: let’s do RAG. Indexing, vector store, retrieval, answer with sources. It has become the default reflex for any question about company knowledge.

The reflex is understandable, but it is not always right. A RAG system adds components, intermediate states and failure points. It can retrieve the wrong document, miss context, return correct but insufficient chunks, add latency and make it harder to understand why the model answered the way it did.

The point is not to choose the newer acronym. It is to understand what kind of knowledge the system is using and what behavior the team expects from it.

When retrieval makes sense

RAG is still a good choice when the knowledge base is large, changes often or needs granular access. If you have thousands of documents, changing policies, tickets, contracts, operational knowledge bases or content with distributed ownership, you do not want to load everything into the prompt every time. You want to retrieve the most relevant subset at the moment of the question.

The documentation for OpenAI File Search and LlamaIndex RAG describes this logic well: the model is not trained on your data, but receives relevant material at runtime. It is a powerful pattern, especially when the system needs to stay current without retraining.

The hidden cost is retrieval quality. If chunks are weak, metadata is poor or queries are ambiguous, the model can look smart while reasoning on incomplete material. Many hallucinations start here: not in generation, but in a fragile retrieval layer. I wrote about this in the article on retrieval failures.

That is why a RAG implementation also needs a retrieval contract. The contract defines source scope, freshness, fallback and ownership before the model starts answering with a confident tone.

When context can stay close to the model

CAG, cache-augmented generation, starts from a different observation: if the knowledge is limited, stable and manageable inside a long context window, perhaps you do not need to retrieve on every request. The paper “Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks” proposes exactly this alternative: preload the relevant resources and use context or inference-state caching to reduce complexity and latency.

Anthropic makes a related point in its article on Contextual Retrieval: for smaller knowledge bases, it can be simpler to include the full material in the prompt, especially when prompt caching makes repeated context more affordable. Google describes a similar principle in the Gemini context caching documentation: when the same input tokens are passed repeatedly, caching can reduce cost and time.

This path works well for small manuals, stable internal procedures, standard contracts, project briefs, operational policies or document sets that do not change every day. It does not remove governance. It moves it: the team needs to know which context version is loaded, when it expires, who updates it and which questions fall outside the system boundary.

The real choice is maintenance versus variability

The practical question is not “RAG or CAG?”. It is: how often does the knowledge change, how large is it, how expensive is a retrieval mistake, how explainable should the answer be and who will maintain the system in three months?

If the knowledge changes often, RAG allows more modular updates. If the knowledge is compact and stable, CAG can remove unnecessary components. If the domain is high risk, the team still needs evaluation, logs and human fallback. No architecture makes a system reliable if nobody measures it.

For a startup or an operating team, starting simple is often the better decision. An assistant over five documents probably does not need a complex RAG platform. An assistant over thousands of procedures cannot live on long prompts alone.

Maturity is not falling in love with the pipeline. It is choosing the smallest piece of architecture that makes the system verifiable, maintainable and useful in daily work.

The decision can be revisited later. Starting with CAG for five stable documents does not prevent moving to RAG when the corpus grows. Starting with RAG for a changing knowledge base does not remove the need to cache stable context. The mature choice is the one the team can maintain without turning answer quality into guesswork.