RAG Reranking

What is RAG reranking?

First-stage retrieval (vector or hybrid search) is built for speed across millions of chunks — and its accuracy shows it. Reranking adds a second stage: take the top 20–100 candidates and rescore each with a cross-encoder, a model that reads the query and the document together and outputs a relevance score. Bi-encoder retrieval compares two independently-computed vectors; the cross-encoder sees the actual interaction between question and text, which is why it reliably catches what cosine similarity misses — negation, specificity, “similar topic but wrong answer.”

The retrieve-wide, rerank-narrow pattern

The standard production shape: retrieve top-50 cheaply, rerank, pass top 3–8 to the LLM. This beats both alternatives — top-5-no-reranker (recall failures) and top-50-straight-to-context (noise degrades answers and multiplies token cost). On BEIR-style benchmarks, adding a reranker is often the single largest retrieval-quality jump available, bigger than switching embedding models.

Cost math that justifies it

A reranker call per query (Cohere Rerank, Voyage, or a self-hosted BGE cross-encoder) costs fractions of a cent and ~50–200 ms. Compare the alternative: stuffing 40 extra marginal chunks into a frontier-model context costs more per query and lowers answer quality. Reranking is that rare optimization that improves quality and cost simultaneously — its latency is the only genuine trade-off, which is why latency-critical autocomplete skips it and RAG QA shouldn’t.

What people get wrong

• Reranking the wrong pool. If recall failed in stage one, the reranker sorts garbage; it fixes precision, not recall — widen retrieval first.
• Ignoring chunk-length limits: cross-encoders have token caps; oversized chunks get truncated and silently misscored.
• Skipping evals: reranker gains vary by domain; measure nDCG/recall on your labeled set, not the vendor’s benchmark.

Primary source: Passage Re-ranking with BERT (Nogueira & Cho, 2019)

Where Reranking Sits in a Modern RAG Stack

A typical production RAG pipeline today runs three progressively more precise and expensive stages: broad initial retrieval (hybrid vector plus keyword search across the full corpus, returning perhaps 50-100 candidates), reranking (a cross-encoder rescoring those candidates down to the top 3-10), and finally generation (the LLM producing an answer using only the reranked, most relevant context). Skipping the reranking stage and passing all initially retrieved candidates directly to the LLM is a common shortcut in early-stage RAG implementations, but it tends to produce measurably worse answer quality once a corpus grows large or diverse enough that initial retrieval alone can’t reliably surface only the genuinely relevant content within its top results.