Chunking strategies

Fixed-size, semantic, and recursive chunking — and how each shapes what your retriever can ever find.

Why it matters

Before a document can be embedded and searched, it has to be cut into pieces — and that single decision quietly caps the ceiling of your entire RAG system. A chunk is the atomic unit of retrieval: whatever answer isn't fully contained inside one chunk simply cannot be retrieved, no matter how good your embedding model or LLM is downstream.

Three common strategies trade off differently:

• Fixed-size (e.g. 500 tokens, no overlap) is cheap and predictable, but it slices sentences and even tables in half with no regard for meaning — a policy clause can end up split across two chunks, so neither chunk alone answers the question.
• Recursive chunking tries fixed-size splits first, then backs off to natural separators (paragraph, then sentence, then word) so it rarely cuts mid-sentence, while still keeping chunks roughly uniform in size for the embedding model.
• Semantic chunking splits on actual topic shifts (headings, paragraph boundaries, or embedding-similarity drops between sentences), so each chunk is a coherent 'unit of meaning' rather than an arbitrary character count — at the cost of variable, sometimes unpredictable chunk sizes.

A useful rule of thumb: start with recursive chunking with ~10-15% overlap (so context isn't lost at the seams), and only move to true semantic chunking once you can measure that boundary errors are actually hurting recall.

Retrieval can only ever return a chunk — so the chunk boundary is the smallest unit of truth your system can surface. If a fact is split across two chunks, no amount of model intelligence downstream can reliably recombine it.

There are three families you will actually use:

Two parameters dominate quality:

• Size. Too small and a single chunk lacks the surrounding context the model needs; too large and you dilute the embedding (the vector becomes an average of many topics) and waste prompt budget.
• Overlap. A 10–20% sliding overlap is the cheapest insurance against the boundary problem: the sentence that explains a clause ends up in both neighbouring chunks, so whichever one is retrieved still carries the meaning.

Worked example. Take a policy doc: Refunds are available for 30 days. followed by Enterprise customers require legal approval for refunds over $10k. A naive 80-char split can put the exception in a different chunk from the rule. Now a query 'can an enterprise customer get a $12k refund?' retrieves the general rule, the model answers 'yes, within 30 days', and you have shipped a confidently wrong answer. Overlap or heading-aware chunking keeps the exception attached.

Chunking is shelving a library. If you shelve a contract's exceptions clause on a different floor from its main clause, the fastest librarian in the world still hands you half the contract. The retrieval model is only as good as the shelving decision you made at ingest time.

• Splitting a rule from its exception. Symptom: answers omit caveats. Fix: heading-aware chunking or parent-child retrieval so the exception travels with the rule.
• One chunk per document. Symptom: embeddings are topic soup, recall collapses on specific questions. Fix: split to coherent sections.
• Tiny zero-overlap chunks. Symptom: the answer sentence is cut in half. Fix: 10–20% overlap.
• Treating tables/code as prose. Symptom: a row is split from its header. Fix: keep tables and code blocks intact as their own chunk.