The two searches you keep confusing.

The first time I sat down to build a real retrieval layer, I did what most people do: picked one method and shipped it. Vector search, obviously — it's the new one, the smart one, the one that understands meaning. Six weeks later I was watching it confidently return documents that didn't contain the acronym the user had literally typed, and I had to unlearn most of what I thought I knew.

They fail differently. That's the whole point.

BM25 is a keyword search. It reads your query, tokenises it, and ranks documents by how often the rare words appear. It has no idea what your query means. It doesn't care. That's a feature.

• BM25 wins on: names, IDs, acronyms, verbatim phrases — the exact-token stuff embeddings quietly fumble.
• BM25 loses on: paraphrase. Search for “AI capabilities” and a document about “machine-learning offerings” is invisible.

Vector kNN is semantic search. It embeds your query into a high-dimensional space and finds the documents whose embeddings live nearby. It understands paraphrase almost natively.

• Vector wins on: meaning. “AI capabilities” and “machine-learning offerings” sit right next to each other.
• Vector loses on: exact tokens. Ask for a specific ID and the embedding shrugs.

Once I saw the failure modes were disjoint, it became obvious that picking one is a choice to be wrong in a specific way. Both methods are cheap. Run both.

Fusing scores doesn't work. Fusing ranks does.

The trap I fell into first: normalise the two score distributions and add them. BM25 lives on a 0-to-50-ish scale; cosine similarity on 0-to-1. You can normalise them, but the moment you do, you're inventing a weighting scheme with no principled way to tune it. Every query has a different mix. Your normalisation drifts. Your relevance drifts with it.

Reciprocal Rank Fusion sidesteps the whole problem by throwing the raw scores away and keeping only the rank position in each list. A document that appears in both lists at high rank gets a strong combined score; a document that only one method liked gets penalised. No tuning knobs. No score normalisation. Just: chunks that look relevant by both meaning and keywords float to the top.

That's the one-line intuition I wish someone had given me on day one.

The reranker is the honesty stage

After fusion, you still have dozens of candidates that individually looked relevant to at least one method. A cross-encoder reranker reads the query and each candidate together (unlike BM25 and vector, which score each side alone) and asks: given the query, does this chunk actually answer it? It catches negation, hedging, and near-miss topical drift.

The reranker's output is the first signal in the pipeline you can trust to reject. Everything before it is optimistic recall. The reranker is where you buy back precision.

I now write a hard confidence floor. If the reranker's score on a chunk is below the floor, the chunk is dropped — no matter how confidently BM25 or the vector search liked it. And a safety minimum: if the floor would leave you with too little context, keep everything, because giving an LLM zero context is worse than giving it noisy context.

The one honest bypass

Person names broke everything. Embeddings would confidently pull paragraphs that mentioned similar-sounding names, and the reranker — which was optimised for topical relevance — would rank them above the actual canonical hit in a role field. I lost a day to this.

The fix wasn't clever. When the query is clearly a person or role lookup, skip the whole hybrid stack and do a plain keyword match against the role field. Plain BM25. No fusion. No reranker. No floor.

The lesson: pipelines that treat every query identically are pipelines that will disappoint you on specific query shapes. Add named bypasses where the general path is provably wrong, and log the fact that you took the bypass so future-you can audit it.

What I'd tell past-me

• Ship both methods on day one. Do not fall in love with either.
• Fuse ranks, not scores. Trust the arithmetic; distrust your intuition about weighting.
• Add a reranker; add a floor; add a floor-safety minimum. In that order.
• Watch what queries fail. Design bypasses for the ones the pipeline is structurally bad at.
• Log the pipeline's reasoning. Every dropped chunk, every rank, every rerank score. You will need it.

Retrieval isn't one problem. It's five stacked problems that share a shape, and getting each stage's job clear in my head is what finally made the whole thing feel less like magic.