LLM "hallucinations" are often retrieval failures

Some time ago I built an AI version of myself on Slack with RAG, Pinecone, n8n and OpenAI. The model was not the hard part. The hard part was deciding what counted as retrievable knowledge: how to chunk notes, how to tag them, how to handle stale context, and what the assistant should do when search returned something plausible but incomplete.

That work changed how I read many LLM “hallucinations”. Some are model behavior. Many others start earlier. The answer fails because the model sees the wrong evidence, half of the evidence, or no evidence at all and is still invited to answer.

This is why I treat retrieval as a product surface, not only as a technical layer. A retrieval contract has to define what the system is allowed to retrieve, how confidence is judged, and when the assistant must refuse, ask a question, or expose uncertainty.

What breaks before the answer?

The failure usually starts with a quiet assumption: if the document is somewhere in the vector database, the system “knows” it. That is not true in production.

A RAG system has to decide which documents enter the index, which chunks carry enough meaning, which metadata matters, which filters are mandatory, and how much context the model receives. Each choice can distort the answer before generation begins.

If the chunk is too small, the model gets fragments without the decision context around them. If the chunk is too large, the relevant sentence may be diluted inside a broad passage. If metadata is weak, the system may retrieve a good paragraph from the wrong customer, market, time period, product version, or policy. The generated answer then sounds fluent, but the upstream evidence was already compromised.

This is also why “just add a bigger model” is often the wrong reflex. A stronger model can reason better over good context. It cannot reliably reconstruct the missing part of a messy retrieval pipeline.

Why vector search is not enough?

Vector search is useful because it finds semantic similarity. But similarity is not the same as truth, ownership, freshness, permission, or operational relevance.

In real company knowledge, the right answer often depends on relationships and constraints. A sales note may mention a feature, but the source of truth may be the roadmap. A support ticket may mention a workaround, but the durable rule may live in a policy document. A customer quote may be relevant only for one segment. A workflow may depend on who owns an exception.

That is why a pure vector approach can retrieve something that looks close while missing the structure that makes it correct. The same issue appears when deciding RAG or CAG for an operational system: the architecture choice matters less than the decision contract around evidence, latency, and repeatability.

The practical question is not “did search return a result?” The question is: did it return the right evidence for this decision, under the right constraints, with enough context to avoid pretending?

What does AkasicDB get directionally right?

The KAIST and GraphAI work on AkasicDB caught my attention because it points in this direction. The research release describes a system that integrates vector, graph and relational retrieval into one query plan, with a DOI-linked SIGMOD demonstration. The reported numbers, including a large hallucination reduction, should be read carefully because this is research, not a production guarantee.

Still, the direction is useful. Real business knowledge is not only semantic. It is also relational and structured. Customers belong to accounts. Accounts belong to segments. Policies have dates. Product decisions have owners. Metrics have definitions. Permissions matter.

When those dimensions live in separate systems, the application layer has to stitch them together after retrieval. That is exactly where many RAG systems become fragile. They fetch a semantically plausible chunk, then ask the model to compensate for missing structure. The model does what it was asked to do: it completes the pattern.

What should a production RAG checklist include?

For me, the minimum checklist is operational, not academic.

First, every retrievable source needs ownership, date, scope and freshness rules. If nobody owns the source, the model should not treat it as durable truth.

Second, the assistant needs a fallback policy. When the evidence is weak, it should say so. A RAG system without refusal behavior is a confidence machine.

Third, retrieval needs tests. Not only prompt tests, but evidence tests: given this question, which source should be retrieved, which source should be excluded, and what answer should be impossible without more context? This is close to the governance problem I wrote about in production n8n workflows: the workflow is not done when it runs, it is done when the exceptions are visible and owned.

Fourth, teams should inspect failures at the retrieval layer before blaming the model. Was the source missing? Was the chunk wrong? Was metadata absent? Was the relationship needed? Was the model asked to answer despite thin evidence?

Where should we be careful?

I would not turn one paper or one benchmark into a buying decision. The useful lesson is narrower: retrieval quality is not a backend detail. It is part of the user promise.

The OpenAI retrieval guide is a useful reminder that retrieval systems need file selection, search behavior and tool configuration, not just model prompting. The same applies outside OpenAI. If the assistant can touch company knowledge, the team needs to decide what knowledge means.

So when an LLM hallucinates in a RAG product, I try to ask a colder question first: did the model invent something, or did we hand it the wrong room and ask it to describe the whole building?