RAG needs retrieval contracts, not bigger windows

A RAG workflow does not become reliable because the model can swallow more tokens. It becomes reliable when the team can explain which evidence the system is allowed to retrieve, who owns that evidence, how stale it may be, what happens when the evidence is weak, and how quality is tested before the workflow reaches customers or frontline teams.

That is the retrieval contract.

For a product or operations leader, this matters because most RAG failures do not look dramatic at first. The answer sounds plausible. The cited document may exist. The model may even be technically using retrieval. But the workflow still gives the wrong refund policy to support, the old pricing rule to sales, or the obsolete security clause to procurement. The visible symptom is an AI answer. The operational cause is an underspecified retrieval system.

OpenAI’s Retrieval API documentation describes retrieval in terms of semantic search, vector stores, attributes for filtering, ranking options, score thresholds, and hybrid search controls. Those are not just implementation details. They are knobs that should reflect a business agreement about what the workflow is allowed to know and when it should refuse to answer. OpenAI’s Retrieval API documentation is useful because it makes the mechanics explicit: files are chunked, embedded, indexed, filtered, ranked, and returned.

Anthropic’s Contextual Retrieval article adds the other half of the problem: traditional RAG can lose useful document context when information is split into chunks, so teams should improve retrieval with contextualized chunks, BM25 plus embeddings, reranking, and evals rather than assuming a larger prompt will fix the system. Anthropic’s Contextual Retrieval article also notes that longer prompts can be a simple option for smaller knowledge bases, but as the corpus grows, retrieval quality becomes the operating constraint.

The answer is only as good as the retrieved evidence

When an AI workflow is connected to internal knowledge, the model is not the only product surface. The corpus is a product surface. The filters are a product surface. The fallback message is a product surface. The freshness rule is a product surface.

A support copilot answering from a policy library is not simply generating text. It is selecting evidence from a moving system of help-center pages, internal macros, regional exceptions, release notes, and escalation rules. If those sources are mixed without ownership, the copilot may retrieve a public FAQ when it needs a private policy, a global rule when it needs a regional exception, or an archived page because it happens to be semantically similar.

This is why the common fix, “let’s use a larger context window”, is dangerous. More context can reduce some misses, but it can also hide the fact that nobody has decided which documents are authoritative. It can pass more irrelevant text to the model, increase review difficulty, and make errors harder to debug. The team may celebrate fewer empty answers while quietly increasing the number of confident but weakly grounded ones.

This connects directly to the argument in LLM “hallucinations” are often retrieval failures. If the wrong evidence is retrieved, the model can produce a fluent answer for the wrong reality. And as argued in AI automation needs failure classes, not retries, repeating a broken operation is not resilience. A RAG workflow needs to know whether it failed because no source matched, because sources conflicted, because metadata was missing, or because the answer required a human decision.

What belongs in a retrieval contract?

A retrieval contract is a short operating artifact, not a research paper. It should be readable by the workflow owner, the subject-matter owner, and the technical builder. It defines the rules that sit between a user query and the answer.

Start with corpus scope. Which collections are in bounds for this workflow? A customer-support assistant may use approved help-center content, current policy documents, and internal escalation macros. It should not use draft launch notes, Slack exports, or sales enablement decks unless the workflow has a reason to treat them as answerable sources.

Then define source ownership. Every collection needs a named owner, not just a department. If the refund policy is wrong, who can correct the source? If a regional exception expires, who removes it? Without ownership, the RAG system becomes a mirror of organizational ambiguity.

Freshness comes next. Some knowledge can be valid for years, such as brand tone guidance. Some knowledge can expire quickly, such as pricing, promotions, legal notices, or incident status. The contract should define acceptable age by source type and the fallback when a source is too old.

Metadata and filters are not optional. OpenAI’s retrieval documentation shows attribute filtering as a way to narrow results by file attributes before semantic search. Operationally, that means the team should decide which metadata matters: region, product line, customer segment, language, confidentiality level, effective date, lifecycle status, or source type. If the workflow must answer only from Italian policy documents for enterprise customers, that is a contract rule before it is an API parameter.

Ranking and reranking policy should also be explicit. What is the minimum relevance score? How many chunks can be returned? Is hybrid search needed because users search by exact SKU, error code, clause number, or internal acronym? Anthropic’s article is especially useful here because it shows why semantic embeddings alone may not be enough, and why contextual chunks, keyword retrieval, and reranking can matter.

Finally, define fallback behavior. The system should not improvise when evidence is missing. It should say what it could not verify, ask a clarifying question, route to a human, or create a task for the source owner. A fallback is not a failure of the experience. It is the experience choosing not to lie.

Where do bigger context windows stop helping?

Bigger context helps when the correct evidence is available, small enough to include, and easy for the model to use. It helps less when the problem is source governance.

If a document is obsolete, a larger window can include more obsolete text. If a policy conflicts with another policy, a larger window can include both without deciding which one wins. If the corpus lacks metadata, a larger window cannot reliably distinguish public from internal, US from EU, current from archived, or draft from approved. If the chunk boundary cuts away the definition of a term, a larger window may only deliver more fragments.

The better question is not “how many tokens can we fit?” The better question is “what must be true before this workflow is allowed to answer?”

For example, a sales knowledge assistant may need three conditions before it can answer a pricing question: the retrieved document must be from the approved pricing collection, the effective date must cover today’s quote, and the customer segment must match the account. If those conditions are not met, the correct behavior is not a longer answer. It is a refusal, a clarification, or an escalation.

This is the same operating mindset behind RAG or CAG: the operational choice before the architecture. Architecture should follow the knowledge pattern. If the knowledge base is small, stable, and reused often, a cached-context approach may be practical. If the knowledge base is large, changing, permissioned, and full of exceptions, the retrieval contract becomes the center of the design.

How to test retrieval before scaling the workflow

Testing a RAG workflow by reading ten impressive answers is not enough. The team needs an eval set that represents the questions the workflow must survive.

A practical eval set should include normal questions, edge cases, obsolete-source traps, region-specific questions, exact-match lookups, ambiguous terms, and questions the system should refuse. Each test should define the expected source behavior, not only the expected final answer. Did the system retrieve the current policy? Did it avoid the archived page? Did it rank the regional exception above the global default? Did it ask for the missing customer segment?

The retrieval contract gives those evals teeth. Without it, reviewers debate whether an answer “looks good”. With it, they can inspect whether the workflow followed the agreed evidence rules.

A useful audit for one workflow can be simple:

1. List the top 30 real user questions.
2. Map the approved source collections for those questions.
3. Add required metadata fields and freshness rules.
4. Define the ranking and fallback policy.
5. Build an eval set with expected sources, not just expected prose.
6. Review misses by failure class: missing source, stale source, wrong filter, poor chunking, weak ranking, or no fallback.

This is not bureaucracy. It is how an AI automation becomes operable.

Make the contract smaller than the ambition

The best first retrieval contract should cover one workflow, not the whole company. Choose a support escalation flow, a sales enablement assistant, an internal policy bot, or a procurement FAQ. Write the contract for that workflow in plain language. Then tune the retrieval system to honor it.

The thesis is simple: RAG automations need retrieval contracts, not bigger context windows. Larger windows can be useful, but they are not a substitute for source scope, ownership, freshness, filters, ranking, fallback, and evals. If the workflow cannot say what evidence it trusts, it is not ready to answer at scale.