Evaluation · 2026-09-06 · 994 words · 5 min read

Bootstrap CIs for eval deltas: is a 4-point gap on 500 items real?

A paired bootstrap for the accuracy gap between two systems on 500 items, with McNemar's test, coverage checked over 1,000 replays, and the code and per-item data.

statistics bootstrap evaluation python

System B scores 74.8% on your 500-item test set and system A scores 71.2%. Is B better? The honest answer from that sentence alone is that you cannot tell, and this article shows exactly how much you cannot tell, with a simulation where the true gap is known, a paired bootstrap that gets the interval right, and a coverage check that proves it. The listing is code/bootstrap-eval-deltas.py, standard library only; the per-item scores of the reported run are datasets/bootstrap-eval-deltas.csv.

Dror et al. (ACL 2018) surveyed significance testing in NLP papers and found it mostly absent or misapplied; their protocol recommends paired tests when two systems are scored on the same items, and non-parametric methods when the metric's distribution is unknown. The bootstrap here follows that advice. What the paper does not give you is a feel for the numbers, which is what a simulation with known truth provides.

The simulation

Two systems with true accuracies 0.72 and 0.76, so the true gap is +4.0 points. Both are scored on the same 500 items. Item difficulty is shared: a latent Gaussian per item feeds both systems' outcomes with correlation 0.6, via NormalDist.inv_cdf from the standard library's statistics module, so that each system hits its target accuracy while their errors correlate the way real systems' errors do (hard items are hard for both).

def simulate(rng):
    out = []
    w = math.sqrt(1 - CORR ** 2)
    for _ in range(N_ITEMS):
        d = rng.gauss(0, 1)
        za = CORR * d + w * rng.gauss(0, 1)
        zb = CORR * d + w * rng.gauss(0, 1)
        out.append((int(za < _ZA), int(zb < _ZB)))
    return out

That correlation is the whole reason pairing matters, and it is the thing an unpaired analysis throws away.

Four things the script computes

Per-system percentile bootstrap CIs. Resample the 500 items with replacement 2,000 times, compute accuracy each time, take the 2.5th and 97.5th percentiles.

The paired bootstrap CI of the difference. Resample item indices, and for each resample compute B's accuracy minus A's accuracy on the same items. Because both systems' scores on an item stay together, the shared difficulty cancels.

def paired_bootstrap(scores, rng):
    n = len(scores)
    deltas = []
    for _ in range(B):
        idx = [rng.randrange(n) for _ in range(n)]
        deltas.append(sum(scores[i][1] - scores[i][0] for i in idx) / n)
    return pct_ci(deltas)

The unpaired CI of the difference, resampling A's items and B's items independently, to show what ignoring the pairing costs.

McNemar's exact test on the discordant pairs: items where exactly one system is right. Under the null hypothesis that the systems are equally good, the split between B-only-right and A-only-right is binomial with p = 0.5.

def mcnemar_exact(scores):
    b_only = sum(1 for a, b in scores if b and not a)
    a_only = sum(1 for a, b in scores if a and not b)
    n = a_only + b_only
    k = min(a_only, b_only)
    p = sum(math.comb(n, i) for i in range(0, k + 1)) / 2 ** n * 2 if n else 1.0
    return a_only, b_only, min(1.0, p)

The numbers

n=500, true accuracies A=0.72 B=0.76, true gap=+4.0 pts
observed: A=0.712 [0.672, 0.750]   B=0.748 [0.712, 0.786]
observed gap: +3.6 pts
  paired bootstrap 95% CI:   [-1.4, +8.4] pts
  unpaired bootstrap 95% CI: [-1.8, +9.0] pts
  discordant pairs: B-only correct=85, A-only correct=67; McNemar exact p=0.1677

The observed gap is 3.6 points, close to the true 4.0. And the 95% interval on that gap runs from -1.4 to +8.4 points. With 500 items and a true 4-point difference, the data are consistent with B being slightly worse than A. McNemar agrees: 85 versus 67 discordant items gives p = 0.17, nowhere near 0.05.

That is the finding. A 4-point accuracy gap on 500 items is not reliably detectable. If your eval set is 500 items and your improvements are of that size, you are reading noise most of the time.

The unpaired interval is only slightly wider here, [-1.8, +9.0] versus [-1.4, +8.4], because a correlation of 0.6 on the latent scale translates to a modest correlation on binary outcomes. On tasks where the systems agree on most items, the gap between paired and unpaired widths grows, and the paired interval is always the right one when the items are shared.

Coverage: does the interval do what it claims?

A 95% interval should contain the true value 95% of the time. The script checks this by replaying the whole experiment 1,000 times with fresh simulated scores and recomputing the paired interval each time:

over 1000 replays: paired CI covers the true gap 94.4% of the time;
CI excludes zero (a 'significant' +gap) 34.9% of the time

Coverage is 94.4%, which is what a 95% interval should give within Monte Carlo error. The second number is the one to remember: when the true gap is 4 points and n is 500, the interval excludes zero only 35% of the time. That is the statistical power of this experiment. Sixty-five percent of the time you would run this exact comparison and correctly conclude that you cannot tell.

What to do about it

Three levers, in order of how much they help.

More items. Standard error scales with one over the square root of n; detecting a 4-point gap with 80% power at this correlation needs on the order of 2,000 items, not 500. Edit N_ITEMS at the top of the script and rerun the coverage loop to see the power at any size before you build the set.

Pairing, always. Score both systems on the same items and use the paired interval or McNemar. Comparing two systems on two different samples of 500 is strictly worse and a common mistake when eval sets get resampled between runs.

Report the interval, not the point estimate. "B is +3.6 points, 95% CI [-1.4, +8.4]" is a sentence that tells the reader what you know. "B is 3.6 points better" is a sentence that will be wrong about a third of the time on data like this.

The data

datasets/bootstrap-eval-deltas.csv holds the per-item scores of the reported run (seed 2026), so the interval and the McNemar count can be recomputed without the simulator.

_README: - item: 1 to 500 - system_a_correct: 1 if system A scored the item correct, else 0 - system_b_correct: the same for system B

Limitations

The simulation is binary accuracy; for continuous metrics like F1 or a judge score, the same paired bootstrap applies unchanged, but McNemar does not, and the interval widths will differ. The Gaussian copula is one way to make errors correlate, not the way real systems' errors correlate; the coverage result holds regardless, because the bootstrap does not assume the copula, but the power figure is specific to this correlation. And a percentile bootstrap can be slightly off for very skewed statistics at small n; at n = 500 and accuracy near 0.7 it is fine, and the 94.4% coverage is the evidence.