Hybrid retrieval and reranking

Two ranked lists merged by reciprocal rank fusion, rank by rank, then a cross-encoder rereading the top twenty; recall@k for every stage.

Loading the animation…

Concept

BM25 and dense retrieval fail on different questions: one misses paraphrases, the other misses rare exact words. Hybrid retrieval runs both and merges the lists. Their scores are not comparable (a BM25 score of 11 and a cosine of 0.6 mean nothing to each other), so the robust way to merge is by rank: reciprocal rank fusion (Cormack, Clarke and Büttcher, SIGIR 2009) gives each chunk $\sum_r 1/(k + \mathrm{rank}_r)$ , with $k = 60$ from the paper. A chunk near the top of both lists beats one at the very top of only one. The animation adds one rank of both lists per step; when both lists hold the same chunk at a rank, its score doubles.

Over all 200 questions (top 50 of each list fused), RRF ranks the answer's chunk first 81% of the time, against 79% for BM25 alone and 71% for dense alone; its nDCG@10 is 0.912. The alternative, normalising each list's scores to $[0, 1]$ and taking $\alpha \cdot \text{dense} + (1 - \alpha) \cdot \text{BM25}$ , does a little better here at $\alpha = 0.5$ (84%, nDCG@10 0.928), but it has a weight to tune, and the right weight depends on the corpus.

Reranking is the second stage. A cross-encoder reads the question and a chunk together, every word of one attending to every word of the other, and outputs one relevance score. It is far more accurate than comparing two separately computed vectors, and far too slow to run over a whole corpus, so it rereads only the top $n$ of the first stage. Here it is ms-marco-MiniLM-L6-v2, trained on the MS MARCO passage-ranking task; its scores were recorded once, offline, for each question's candidates (6,226 question–chunk pairs). Reranking RRF's top 20 puts the answer's chunk first 89% of the time (nDCG@10 0.952). A reranker cannot find what the first stage missed; it can only reorder, which is why the first stage is tuned for recall at a generous depth and the second for precision at the top.

Maths

For a chunk at ranks $r_1$ and $r_2$ in the two lists, $\mathrm{RRF} = \frac{1}{k + r_1} + \frac{1}{k + r_2}$ . With $k = 60$ , first place in one list and absence from the other scores $1/61 \approx 0.0164$ , while fifth place in both scores $2/65 \approx 0.0308$ : agreement wins. A small $k$ makes the top ranks count for more (with $k = 10$ , first place alone is $1/11$ ); a large $k$ flattens the curve. Only ranks enter, so no calibration between retrievers is needed.

The weighted alternative min-max normalises each retriever's top 50, $\hat s = (s - s_{\min})/(s_{\max} - s_{\min})$ , and gives the rest 0. It is sensitive to outliers: one very high BM25 score squashes every other BM25 score towards 0.

For the metrics: with $R$ the chunks that hold the answer (one, except where chunks overlap), recall@ $k = |R \cap \text{top}_k|/|R|$ ; MRR@10 is $1/\text{rank}$ of the first relevant chunk (0 below rank 10); nDCG@10 $= \sum_{i \le 10} \mathrm{rel}_i/\log_2(i + 1)$ divided by the same sum for the ideal ranking.

Code

Reciprocal rank fusion and the rerank, cut from the vendored engine (retrieval.ts):

export function rrf(rankings: number[][], n: number, k = RRF_K, depth = 50): number[] {
  const s = new Array<number>(n).fill(0);
  for (const r of rankings) {
    for (let pos = 0; pos < Math.min(depth, r.length); pos++) s[r[pos]!] = s[r[pos]!]! + 1 / (k + pos + 1);
  }
  return s;
}

const sorted = [...top].sort((a, b) => ce.get(b)! - ce.get(a)! || pos.get(a)! - pos.get(b)!);
return sorted.concat(order.slice(n));