Chunking

One text cut three ways, fixed, recursive and semantic: where the boundaries fall, which answers they cut in two, and what each chunking does to recall.

Loading the animation…

Concept

Before anything can be retrieved, documents are cut into chunks, and the chunk is the unit everything else sees: BM25 counts words per chunk, the embedder makes one vector per chunk, and the window receives whole chunks. Where the cuts fall decides whether the answer to a question sits in one chunk, two chunks, or a chunk so long that its vector is a blur. Three chunkers, all cutting only between the tokenizer's pieces so that sizes are exact Qwen2.5 token counts:

Fixed: fill each chunk to $S$ tokens and cut, wherever that falls, mid-sentence if need be. Optionally overlap neighbours by a few tokens so that a sentence cut at one boundary appears whole in the next chunk.
Recursive (the scheme of LangChain's RecursiveCharacterTextSplitter): split at paragraphs; split a paragraph that is too long at sentences, a sentence that is too long at fixed windows; then merge neighbouring pieces while they fit in $S$ .
Semantic: embed every sentence, measure how similar each sentence is to the next (the line along the bottom of the animation), and break where the similarity drops to the article's lowest quarter, as well as wherever $S$ would be exceeded.

The animation lays the three chunkers' chunks over one stretch of the complexity-theory article. A question asked about this stretch has the answer "set of triples"; fixed 256-token chunks cut it in two (the hatched mark), so no single chunk holds it and that question is lost whatever the retriever does.

The table measures each chunking over all 200 questions. Fixed chunks lose answers at their boundaries: 5 at 128 tokens, 4 at 256, 1 at 512. Long chunks have a second problem: the embedder reads only 256 word pieces, and 73 of the 77 512-token chunks are longer than that, so their vectors describe only their first half; dense recall@5 falls to 0.755, while BM25, which reads every word, gets 0.975. The recursive chunker respects paragraphs, loses no answer, and gives the best dense recall@5 (0.950) with chunks of 180 tokens on average. The semantic chunker makes many small chunks (351, 107 tokens on average) and does no better than fixed 256-token chunks here (dense recall@5 0.900 against 0.900): its breaks follow topic shifts, but small chunks give BM25 and the embedder less to go on, and it still cut 1 answer in two. Overlap (fixed 256 with 64 tokens shared) loses no answer, but some answers now sit in two chunks, which halves recall@1 for those questions by definition; its MRR, which only asks where the first relevant chunk is, is BM25's best of the fixed chunkings (0.854 against 0.829).

Maths

Let a document be a sequence of pieces with token counts $t_j$ . A fixed chunker with size $S$ and overlap $O$ emits chunks $[a, b)$ with $\sum_{a \le j < b} t_j \le S$ (a single piece larger than $S$ is a chunk of its own) and starts the next chunk at the smallest $a' > a$ with $\sum_{a' \le j < b} t_j \le O$ .

An answer of length $\ell$ placed uniformly at random is cut by a fixed chunk boundary with probability about $(\ell - 1)/\bar c$ , for chunk length $\bar c$ in the same units: longer chunks cut fewer answers, overlap of at least $\ell$ cuts none. The semantic chunker's threshold is the nearest-rank percentile $\theta_p = s_{(\lfloor (n - 1) p / 100 \rfloor)}$ of the article's $n$ neighbouring-sentence similarities sorted ascending, computed from the int8 sentence vectors, so the breaks too are bit-identical in both languages.

Code

The fixed chunker's inner loop and the semantic chunker's threshold, cut from the vendored engine (chunking.ts):

while (b < n && (t + pieces[b]![2] <= size || b === a)) {
  t += pieces[b]![2];
  b += 1;
}

const thr = srt[Math.floor(((srt.length - 1) * pct) / 100)]!;
return [thr, sims.map((s) => s <= thr)];