Chapter 40 of 57

Chunking Strategies, on One Page

Created Jul 5, 2026 Updated Oct 27, 2026

Before a retrieval system can find anything, someone has to slice the documents into pieces to embed and index. That slicing — chunking — feels like plumbing, so it's where teams spend the least thought and lose the most quality. Get it wrong in a particular way — the evidence an answer needs never lands in any retrieved chunk — and no reranker or bigger model downstream can put it back.

The whole game is one tension:

Chunks too large — each chunk sprawls across several topics, its embedding becomes a muddy average of all of them, and the specific passage that answers the query gets diluted. Retrieval turns fuzzy, and you stuff the context window with padding.
Chunks too small — you cut a thought in half. The sentence that says "customers are eligible for a refund" lands in one chunk; the next sentence, "…only within 30 days of purchase," lands in another. Retrieval surfaces the first, the model reads a confident yes, and the qualifying condition never made it into the context. How badly that bites depends on the rest of the pipeline: retrieving several neighbours instead of one, or indexing small pieces but handing the generator the parent section, can put the condition back.

An insurance policy paragraph with internal dependencies — an answer in one sentence, the condition that gates it in the next. Drag the chunk-size slider: at large sizes, each chunk holds both the answer and the condition; at small sizes, the answer is in one chunk and the condition in another. For a sample query, the simulator shows which chunk gets retrieved, and whether the LLM ends up looking at the complete rule or just the half that says "yes."

The strategies, roughly in order of how much structure they respect:

Fixed-size — cut every N tokens. Trivial to implement, and it happily slices through the middle of sentences and tables. Also a stubbornly competitive baseline, which is the part people do not expect.
Recursive / structure-aware — split on the document's own seams first (headings, paragraphs, sentences), only falling back to hard cuts when a piece is still too long. Keeps coherent units intact.
Semantic — start a new chunk where the meaning shifts, detected by a drop in embedding similarity between adjacent sentences. The most faithful to topic boundaries in principle, clearly the most expensive to compute, and not reliably better at retrieval than the cheap options.
Overlap (sliding window) — not a fourth strategy but a knob on any of the three: let consecutive chunks share a margin of tokens. It helps a fact that straddles one boundary, does nothing for a dependency longer than the margin, and costs duplicated tokens in the index and in the results.

There's no universal winner: the right chunk size depends on your documents, your embedding model's context limit, and the shape of the questions you expect. The honest way to choose is to hold the corpus, the queries and the retrieval pipeline fixed, and compare evidence coverage and answer quality at comparable context budgets — not just recall at a fixed number of chunks, since five 800-token chunks hand the model eight times the context of five 100-token ones — which is exactly what the Chunking Strategies lab does, on real text. The full note is Chunking Strategies.