Hybrid search in Python: BM25 plus embeddings, measured

Hybrid search runs a keyword ranker and an embedding ranker side by side and merges the two lists. There are two common ways to merge them: reciprocal rank fusion (RRF), which adds reciprocal ranks and has nothing to tune, and a weighted blend of normalised scores, which has one weight to set. Bruch, Gai and Ingber (arXiv 2210.11934v2) argued that the blend beats RRF and fixed its weight at 0.8 on the embedding side. This tutorial builds both in one listing, code/hybrid-search-bm25-dense-fusion.py, and measures them on BEIR's SciFact set with BM25 and two small embedding models. With all-MiniLM-L6-v2, the model the paper used, its result reproduces: the 0.8 blend scores 0.754 nDCG@100 on the 300 test claims against the paper's 0.753, and beats RRF by 0.017 nDCG@10 with a bootstrap interval clear of zero. With bge-small-en-v1.5, a stronger model, the same 0.8 scores below RRF and no better than the embedding model alone. The weight has to move to 0.95, chosen on training claims, and even then the blend's lead over untuned RRF is inside the noise. Every number below comes from a run of the listing, and the per-claim scores are in datasets/hybrid-search-bm25-dense-fusion.csv.

Two ways to merge two rankings

RRF comes from Cormack, Clarke and Büttcher's SIGIR 2009 paper. A document's score is the sum, over the rankers, of 1/(k + r), where r is its rank in that ranker, "where k = 60 was fixed during a pilot investigation and not altered during subsequent validation". The pilot "indicated that k = 60 was near-optimal, but that the choice was not critical", and the constant "mitigates the impact of high rankings by outlier systems". RRF never looks at a score, which is its appeal. A BM25 score runs from 0 to whatever the query's rare words add up to, a cosine similarity sits between -1 and 1, and ranks need no reconciling.

The blend does look at scores: alpha times the normalised embedding score plus (1 - alpha) times the normalised BM25 score. Bruch et al.'s normalisation, which they call TM2C2, subtracts each score's theoretical minimum (0 for BM25, -1 for cosine) and divides by the best score among the candidates (their equation 4), so each ranker's top document gets 1. Their abstract states four findings: RRF is "sensitive to its parameters"; once the blend is tuned, the choice of normalisation barely matters; the blend "outperforms RRF in in-domain and out-of-domain settings"; and it is "sample efficient, requiring only a small set of training examples to tune its only parameter". They tuned alpha on the validation sets of their in-domain collections, "found that α = 0.8 works well for the three datasets", and kept 0.8 for the zero-shot sets, SciFact among them.

The fusion step in the listing, verbatim:

def prepare(lex, sem):
    """the candidate union of both top-POOL lists, and each retriever's exact 1-based rank of every document"""
    lo, so = ranking(lex), ranking(sem)
    rl, rs = np.empty_like(lo), np.empty_like(so)
    rl[lo], rs[so] = np.arange(1, len(lo) + 1), np.arange(1, len(so) + 1)
    return np.union1d(lo[:POOL], so[:POOL]), rl, rs


def fuse(lex, sem, method, param, norm="tmm", prep=None):
    """lex, sem: whole-corpus scores for one query. Returns fused scores (-inf outside the candidate union)."""
    cand, rl, rs = prep or prepare(lex, sem)
    fused = np.full(lex.shape, -np.inf)
    if method == "rrf":
        fused[cand] = 1 / (param + rl[cand]) + 1 / (param + rs[cand])
    else:
        l, s = lex[cand], sem[cand]
        if norm == "tmm":
            ln, sn = l / l.max(), (s + 1) / (s.max() + 1)
        else:
            ln, sn = (l - l.min()) / (l.max() - l.min()), (s - s.min()) / (s.max() - s.min())
        fused[cand] = param * sn + (1 - param) * ln
    return fused

The candidates are the union of each ranker's top 1,000, as in the paper. SciFact is small enough to score every abstract for every claim, so the ranks are exact; the paper, working on collections of millions of documents, approximated them inside the union.

The setup

SciFact (Wadden et al., arXiv 2004.14974v6) is "1,409 scientific claims verified against a corpus of 5,183 abstracts", built from the S2ORC corpus. BEIR (Thakur et al., arXiv 2104.08663v4) turned it into a retrieval task: the claim is the query, and an abstract is relevant if it supports or refutes the claim. BEIR's Table 1 lists 300 test queries, 5,183 documents and 1.1 relevant documents per query. We downloaded the Hugging Face copies at pinned revisions and counted 809 training claims with 919 judgments and 300 test claims with 339. BEIR's table gives 920 training pairs, one more than the file holds.

Every tuned setting was chosen on the 809 training claims, and the 300 test claims chose nothing. nDCG@10 is BEIR's headline metric; nDCG@100 and Recall@100 are what Bruch et al. report for SciFact. With about one relevant abstract per claim, nDCG@10 behaves much like the reciprocal rank of the first hit, with a log2 discount in place of 1/rank.

The baselines land where the papers put them

A fusion result means little unless each ranker first reproduces its published number.

bge matches its card to four decimals. MiniLM sits within 0.007 of the paper. Our BM25 lands 0.016 above BEIR's and 0.011 above Bruch et al.'s, and the tokenizer and stemmer are the likely difference: the same listing without stemming scores 0.661 nDCG@10, 0.020 below the stemmed run. How words are cut moves a BM25 baseline by as much as some published fusion gains, which is reason enough to rerun the baseline rather than copy one.

What fusion buys

With MiniLM, the paper's SciFact row reproduces. The 0.8 blend scores 0.754 nDCG@100 against its 0.753, RRF scores 0.740 against its 0.730, and both hybrids clear either ranker alone. Recall@100 is where RRF does best: 0.965 with either embedding model, above the 0.8 blend's 0.952 and 0.948. Plain min-max normalisation, once tuned, lands within 0.002 of TM2C2 for both models, as the paper said it would, although it picks a different weight (0.60 against 0.75 for MiniLM).

The weight that does not transfer

For MiniLM the training curve peaks at 0.75 and the test curve agrees, with 0.8 one step past the top. For bge the curve is a different shape. It stays below RRF until alpha passes 0.8, peaks at 0.95 on the training claims (0.90 on test, 0.742 against 0.740), and the paper's 0.8 lands on its steep flank at 0.720. A stronger embedding model wants more weight, and a weight tuned for one model is a guess for another.

Paired bootstrap on the 300 test claims, 10,000 resamples, nDCG@10. The "best single ranker" is the one with the higher training score: BM25 for the MiniLM pair, bge for the bge pair.

For MiniLM every hybrid beats BM25 with room to spare, and both blends beat RRF. For bge only the tuned blend clears the embedding model alone, by 0.027, and its 0.010 lead over RRF has an interval that crosses zero. The paper's finding holds where the paper measured it and shrinks to a maybe one model further on.

How many labelled claims the weight needs

We tuned the blend weight and the RRF constant on random subsets of the training claims, 200 draws per size, and scored each choice on the test claims. RRF at k = 60 needs no labels and scores 0.715 (MiniLM pair) and 0.730 (bge pair).

The median supports "sample efficient": by 20 claims the median blend is at its full-training value for bge and within 0.006 of it for MiniLM. The 5th percentile tells the rest. With 5 or 10 claims, at least one draw in twenty picks alpha = 0, which is BM25 alone at 0.681, and for bge the blend's 5th percentile stays below untuned RRF until 200 claims. With a few dozen labels, a blend is the better bet on average and the worse one often enough to matter.

Tuning RRF does not pay here

Both pairs' training claims prefer a tiny constant, k = 1 for MiniLM and k = 2 for bge, the opposite end of the range from 60. On the test claims that choice is worth −0.003 and +0.007 nDCG@10, with neither interval clear of zero. Across all fifteen values from 1 to 1,000, test nDCG@10 spans 0.709 to 0.720 for MiniLM and 0.729 to 0.737 for bge. Bruch et al. found RRF sensitive to its parameter on their collections. On this set, its effect on the result is about 0.01, against a swing of 0.645 to 0.734 across the blend weight for MiniLM. Cormack et al.'s "not critical" describes SciFact better.

One claim, both rules

Test claim 660 reads "Ivermectin is used to treat onchocerciasis." After tokenising and stemming, the word "ivermectin" occurs in none of the 5,183 abstracts, and "onchocerciasi" occurs in exactly one, with an idf of 8.148. That abstract is the relevant one. BM25 ranks it first with a score of 8.698, well clear of the runner-up at 5.719. bge ranks it 59th: its cosine is 0.60564, against a top cosine of 0.66137.

With the tuned blend at alpha = 0.95, the relevant abstract gets 1 on the BM25 side and (0.60564 + 1) / (0.66137 + 1) = 0.9665 on the embedding side, so 0.95 × 0.9665 + 0.05 × 1 = 0.968. The strongest rival, BM25's 31st and bge's 7th, gets 4.7345 / 8.6981 = 0.5443 and 1.63421 / 1.66137 = 0.9837, so 0.95 × 0.9837 + 0.05 × 0.5443 = 0.962. The relevant abstract stays first. Even a 5% weight carries BM25's gap, because the two abstracts' cosines differ by 0.029 while their BM25 scores differ by almost 4.

RRF sees ranks 1 and 59, and gives 1/61 + 1/119 = 0.02480. An abstract ranked 4th by BM25 and 14th by bge gets 1/64 + 1/74 = 0.02914, and three others also score higher, so the relevant abstract drops to 5th. This is the paper's third finding in miniature: RRF throws away how far ahead a ranker's top result is, and on a claim decided by one rare word, that distance is the signal.

Where the two rankers disagree

On the test claims, BM25 and MiniLM share 3.30 of their top 10 abstracts on average, and BM25 and bge share 3.50. For 24 claims only BM25 puts a relevant abstract in its top 10 and for 17 only MiniLM does; against bge the split is 12 and 21. The tuned MiniLM blend beats BM25 on 67 claims, ties on 225 and loses on 8; the tuned bge blend beats bge on 52, ties on 216 and loses on 32. An oracle that picks the best weight for each test claim would reach 0.784 and 0.797, against 0.734 and 0.740 for one weight per dataset, so most of what a per-query weight could add is still on the table.

What this does not show

• One small dataset. 300 test claims with about one relevant abstract each give intervals about ±0.02 wide. A difference of 0.01 needs a larger set before it is a finding.
• Possible overlap with MiniLM's training data. Its model card lists S2ORC citation pairs and S2ORC title-abstract pairs among its training sets, and SciFact's abstracts were sampled from S2ORC and its claims written from S2ORC citation sentences. We did not check whether SciFact abstracts were among those pairs. The contamination article shows how one would.
• bge's training split may be easier for it. bge's training-claim score is 0.034 above its test score, against 0.011 for BM25 and 0.015 for MiniLM. The BGE paper (arXiv 2309.07597v5) describes its English data as 200M text pairs and does not name the source datasets, so we cannot say whether SciFact's training claims were among them.
• Choices that differ from the papers. One BM25 field instead of BEIR's two, Porter2 stemming instead of PISA's, ties broken by corpus order. No reranker, and no latency or index-size measurement: everything ran on a 32-thread CPU without a GPU, 435 seconds the first time and about 53 seconds once the embeddings were cached.
• Licence. SciFact is CC BY-NC 2.0 (BEIR Appendix E). The CSV holds our scores and no claim or abstract text, and the listing downloads the data itself.

What to do with it

With no labelled queries, use RRF at k = 60. It was never the best setting on SciFact, but it was never far from the best, it gave the highest Recall@100 of the untuned options, and it cannot be mis-set. With fifty or more labelled queries from your own traffic, tune the blend weight with TM2C2 normalisation, and tune it again whenever the embedding model changes: the 0.8 that suited MiniLM cost bge 0.020 against its own best weight. Judge either choice on a held-out set with an interval, as in bootstrap intervals on eval deltas, because most of the gaps in this article are of the size that 300 queries cannot settle. If you are starting from the TF-IDF index in the minimal retrieval pipeline, swapping in BM25 with stemming and blending one embedding model is the next step, and the self-retrieval check there is still the first test to run.

Running it

pip install numpy==2.3.5 torch==2.12.0 transformers==4.56.2 huggingface_hub==0.36.2 pyarrow==24.0.0 snowballstemmer==3.0.1 matplotlib==3.10.9
python code/hybrid-search-bm25-dense-fusion.py --csv hybrid-search-bm25-dense-fusion.csv --figure fusion.png

The listing reads huggingface.co's robots.txt with its own user agent before downloading anything, then fetches SciFact and the two models at the pinned revisions into --cache and caches the embeddings there. A second run prints the same tables to the last digit.

datasets/hybrid-search-bm25-dense-fusion.csv has one row per claim and system, 1,109 claims by 14 systems, 15,526 rows.

_README:

• split: train (809 claims) or test (300 claims)
• query_id: the SciFact claim id
• n_relevant: the number of abstracts judged relevant to the claim
• system: bm25, bm25_nostem, minilm, bge, and for each of minilm and bge: rrf60_, tm2c2_a0.8_, tm2c2_tuned_ (alpha 0.75 or 0.95), minmax_tuned_ (alpha 0.60 or 0.80) and rrf_tuned_ (k 1 or 2), followed by the model name
• ndcg_at_10, ndcg_at_100, recall_at_100: the claim's scores for that system, binary relevance