Theory
Retrieval is a routing decision
RAG is not one algorithm and it is not the right first move for every question. A production system should classify the information need, select a retriever, and record why that route was chosen. Treating every query as dense top-k search hides exact matches, relationships, and authoritative structured values.
| Query signal | Primary route | Why |
|---|---|---|
| IDs, error codes, quoted phrases, rare names | BM25 / lexical | Exact tokens carry the meaning |
| Paraphrase or conceptual similarity | Dense vector | Meaning matters more than wording |
| Entity relationships or multi-hop paths | Knowledge graph / GraphRAG | The answer is in edges and paths |
| Counts, balances, current status, governed metrics | SQL / metric API | Structured source is authoritative |
| Broad mixed evidence | Hybrid + reranker | Fuse lexical and dense candidates, then rerank |
| Greeting, transformation, supplied-context task | No retrieval | Retrieval adds latency and noise |
A router you can test
Start with deterministic signals: identifier regexes, known entity linking, question type, and available source capabilities. Return a route plus confidence. Low-confidence cases fan out to BM25 and dense retrieval; combine ranks with reciprocal rank fusion, rerank, and keep provenance. Graph retrieval should be explicit, not the default merely because a graph exists.
Evaluate the router separately from answer generation. Build a labelled query set with expected route, expected evidence, and abstention cases. Measure route accuracy, evidence recall, latency, and cost. An answer can sound correct while the router silently chose stale or non-authoritative evidence.