Dense retrieval

Chunks and questions as 384-dimensional embeddings: cosine similarity, nearest neighbours on a 2-D projection, and how much recall survives int8, int4 and one-bit vectors.

Loading the animation…

Concept

A dense retriever turns text into a vector so that texts with similar meaning point in similar directions, then finds the chunks whose vectors are closest to the question's. The model here is all-MiniLM-L6-v2: a six-layer MiniLM transformer, fine-tuned by the Sentence-Transformers project on over a billion pairs of related sentences so that a pair's vectors end up close (its model card, accessed 2026-10-08). Its vector is the average of its last layer's token vectors, normalised to length one: 384 numbers. It reads at most 256 word pieces; anything longer is cut off, which matters for chapter 5.

The model ran once, offline, over every chunk and every question. The site ships the vectors, rounded to 8-bit integers, and the engine finds neighbours by cosine similarity in all 384 dimensions, here in your browser. The picture squeezes those dimensions onto the two directions in which the chunks vary most (principal component analysis, computed offline); those two directions hold only 11% and 9% of the variance, so the articles separate into rough clouds but a neighbour can look far away on the page while being close in 384 dimensions.

On the 200 questions, int8 dense retrieval puts the answer's chunk first 71% of the time and in the top five 95%: behind BM25 (79% and 99%) on this corpus, whose questions borrow the paragraph's words. Where dense retrieval earns its keep is a question phrased differently from the text, as the Moon-landing query of chapter 2 was; chapter 4 combines the two.

How small can the vectors be? Switch the precision. The float32 vectors (1,536 bytes each) were measured once, offline: recall@5 0.950. Rounded to int8 (384 bytes): 0.950, no measurable loss. To int4 (192 bytes, values −7 to 7): 0.935. To one bit per dimension, the sign (48 bytes, compared by counting agreeing bits): 0.870, and the answer's chunk is first only 60% of the time. A 32-fold saving costs real recall at this size; production systems use binary vectors to shortlist and rescore the shortlist at higher precision.

Maths

With $\mathbf{q}, \mathbf{d} \in \mathbb{Z}^{384}$ (each component in $[-127, 127]$ ), $\mathbf{q}\cdot\mathbf{d}$ , $\lVert\mathbf{q}\rVert^2$ and $\lVert\mathbf{d}\rVert^2$ are exact integers below $2^{23}$ , and their product below $2^{53}$ , so the cosine is one correctly rounded square root and one division: the Python reference and this page agree to the last bit.

Scaling a vector does not change its cosine with anything, so each vector gets its own scale ( $127/\max_i |x_i|$ ). Rounding adds an error of at most half a step per component; with 384 components the errors mostly cancel in the dot product, which is why int8 matches float32. The int4 vectors are $\operatorname{sign}(q)\,\lfloor (14|q| + 127)/254 \rfloor$ , rounding $7|q|/127$ half up. The binary similarity $(d - 2h)/d$ , with $h$ the Hamming distance, is the cosine of the two sign vectors; it throws away every magnitude.

Code

Cosine and the int4 re-quantisation, cut from the vendored engine (vectors.ts):

export function cosine(a: Vec, b: Vec): number {
  const na = dot(a, a);
  const nb = dot(b, b);
  if (na === 0 || nb === 0) return 0;
  return dot(a, b) / Math.sqrt(na * nb);
}

const m = Math.floor((Math.abs(q) * 14 + 127) / 254);
out[i] = q < 0 ? -m : m;