Overview
Hallucination, stale chunks, and lost context — the three failure modes that account for almost every bad RAG answer.
Why it matters
Once you've shipped a RAG system, the failures cluster into a small, recognisable set. Naming them correctly is what turns a vague 'the bot is dumb' bug report into a fixable engineering task:
- Hallucination — the LLM generates a fluent, confident answer that isn't actually supported by the retrieved chunks. This often happens when top-k retrieval returns nothing relevant, but the LLM answers from its own parametric memory anyway instead of admitting it doesn't know. Fix: instruct the model to say 'I don't know' when the context is insufficient, and measure this explicitly in evaluation.
- Stale chunks — the vector store still holds an embedding of a document that has since been edited or deleted upstream (e.g. a return-policy PDF was updated last month, but nobody re-ran the ingestion pipeline). The retriever confidently returns the old chunk because nothing tells it the chunk is outdated — the embedding itself carries no notion of 'as-of' time. Fix: incremental reindexing triggered by source-document changes.
- Lost context — the answer needs information from two different chunks (or two different parts of one long chunk), but top-k retrieval only surfaced one of them, or chunking split the relevant sentence away from the sentence that qualifies it. Fix: better chunking, larger k, or moving to a retrieval method that can compose facts across chunks (GraphRAG, Level 2).
Notice these three are almost always retrieval bugs wearing a generation costume — the LLM did exactly what you'd expect with the (wrong, missing, or outdated) context it was handed.