How many eval examples do you need to trust a 3-point gain?

A fine-tuned model scores 74.1% on your eval; production scores 71.0%. Is that a win? With 300 items, our simulation says a true 3-point gain is detected less than one time in five. Most teams size eval sets by how many items they could label in a week, then read differences of a few points as progress. This post gives the arithmetic to size them properly, and a simulation to check the arithmetic.

The variance of an eval score#

Treat each eval item as a draw from the population of inputs you care about (Miller, 2024). If model A answers item $j$ correctly ($c_{Aj}=1$) with overall accuracy $p$, its score over $n$ items has standard error $\sqrt{p(1-p)/n}$. At $p=0.7$ and $n=300$ that is 2.6 points, so a 95% interval is about ±5.2 points — on one model.

Comparing two models is what you actually do, and there are two ways to estimate the uncertainty of the difference. The unpaired estimate treats the two scores as independent:

But both models answer the same items, and items differ in difficulty, so their correctness is positively correlated. Working with per-item differences $d_j=c_{Bj}-c_{Aj}$:

The covariance term is the whole point. The more the two models agree on which items are hard, the more of the noise cancels.

Minimum detectable effect#

For a two-sided test at $\alpha =0.05$ with 80% power, the smallest true difference you can reliably detect is

Inverting gives the eval size you need for a target effect: $n\approx 7.85\mathrm{Var}(d)/{\mathrm{MDE}}^2$. In our simulated setting $\mathrm{Var}(d)=0.298$, so detecting a 3-point gain needs roughly $7.85\times 0.298/0.03^2\approx 2,600$ items. Halving the target effect quadruples the eval.

Simulation#

We simulated eval sets with an item-response model, so that items have real difficulty structure rather than independent coin flips.

• Item difficulty $b_j\sim \mathcal{N}(0,1.5^2)$. Model $m$ answers correctly with probability $\sigma ({\theta }_m-b_j)$.
• Model A is calibrated to 70% population accuracy; model B to 70% + Δ for Δ ∈ {1, 2, 3, 5, 8} points.
• For each $(n,\Delta )$, 4,000 independent eval sets. We record how often an unpaired z-test and a paired test on $d_j$ reject "no difference" at α = 0.05.

The correlation between the two models' per-item correctness came out at about 0.27 — modest, typical of two models of similar strength on a mixed-difficulty set. Real evals of a base model against its own fine-tune tend to be more correlated than this, which makes pairing even more valuable.

Sizing table#

All values in accuracy points. Power columns use the paired test. MDE is the analytic value at 80% power, using the variance measured in simulation.

The headline numbers: with 300 items, the smallest reliably detectable gain is about 8.8 points. With 1,000, about 4.8. You need roughly 2,000–5,000 items before 2–3 point improvements are routinely distinguishable from noise. Pairing buys a constant factor — around 15% narrower intervals here — which is worth taking but does not replace a bigger eval.

When items aren't independent#

The formulas above assume each item is an independent draw. Real eval sets often aren't: ten questions written about the same contract, five turns of the same conversation, twenty SQL questions against one schema. Items in a cluster share difficulty beyond what the model's overall accuracy explains, so they carry less information than their count suggests. With clusters of size $m$ and intra-cluster correlation ${\rho }_c$, the effective sample size is

The denominator is the design effect. With $m=10$ and a modest ${\rho }_c=0.2$, it is 2.8: a 1,000-item eval built from 100 documents behaves like about 360 independent items, and its minimum detectable effect is about 1.7× wider than the naive calculation says. Miller (2024) recommends clustered standard errors for exactly this case. The practical fix is to report and test at the cluster level, or to build evals from many sources with few items each.

Judged metrics need even more items#

Everything above assumes correctness is known exactly. When an LLM judge scores the outputs, its errors add noise and also shrink the measured difference between models by a factor $J=S_e+S_p-1$ (derived in our judge-agreement note). Since required $n$ scales with $1/{\Delta }^2$, a judge with $J=0.8$ needs about $1/0.8^2\approx 1.56\times$ the items to detect the same true gain, and a lenient judge with $J=0.57$ needs about $3.1\times$. A 2,000-item eval scored by a mediocre judge has roughly the resolving power of a 650-item eval scored exactly.

First-order approximation: assumes judge errors are independent of which model produced the output. Judges that prefer one model's style violate this and can bias the difference, not just shrink it.

What to do when you can't label 3,000 items#

1. Decide the effect size you care about first. If a 2-point gain isn't worth shipping, don't build an eval designed to detect one. Write the MDE into the eval spec.
2. Always pair. Score both models on identical items and analyse per-item differences. It costs nothing.
3. Reduce per-item variance. Graded scores (0–1 partial credit) and multiple samples per item at temperature > 0, averaged, both lower $\mathrm{Var}(d)$. Miller (2024) gives the formulas for clustered and resampled designs.
4. Spend labels where models disagree. Items both models get right or both get wrong contribute nothing to $d$. Prioritise labelling and review on disagreement items.
5. Report intervals, not points. "74.1% vs 71.0% (Δ +3.1, 95% CI −3.1 to +9.3, n = 300)" tells a reviewer the truth. Release gates in POFE use the lower bound of the paired interval, not the point estimate.

Related: Evaluating fine-tuned models (the pillar guide), what Cohen's κ hides about LLM judges, and the release-gate checklist.

Frequently asked questions#

References#

1. Miller, E. (2024). Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations. arXiv:2411.00640. arxiv.org/abs/2411.00640
2. Efron, B. (1979). Bootstrap Methods: Another Look at the Jackknife. The Annals of Statistics 7(1), 1–26. doi.org/10.1214/aos/1176344552

Related reading