LLM-as-a-judge agreement: what Cohen's κ hides

The standard advice for LLM-as-a-judge is to label a few hundred outputs by hand and report Cohen's κ between the judge and the humans. That is necessary and not sufficient. κ depends on how many outputs pass, not only on how good the judge is; and a judge with respectable κ can still move your headline pass rate by 20 points and hide most of the gap between two models. This post works through the algebra and puts numbers on it.

A two-number model of a judge#

Treat the human label as ground truth and describe a binary judge by two rates: sensitivity $S_e$, the probability it passes an output humans pass, and specificity $S_p$, the probability it fails an output humans fail. Let $\pi$ be the true pass rate of the outputs being judged. Then

where $p_o$ is observed agreement and $q$ is the pass rate the judge reports. Cohen's κ corrects agreement for chance (Cohen, 1960):

Every quantity below follows from these lines. We evaluate them in closed form for three archetypal judges — balanced ($S_e=S_p=0.90$), lenient ($S_e=0.97,S_p=0.60$) and strict ($S_e=0.75,S_p=0.97$) — and check sampling variability by simulation.

Same judge, different κ#

The balanced judge agrees with humans 90% of the time at every prevalence. Its κ does not stay put: 0.80 when half the outputs pass, 0.72 at 80% and 0.43 at 95%. This is the first of the two paradoxes Feinstein and Cicchetti (1990) described for clinical raters: high agreement, low κ, because chance agreement $p_e$ rises when almost everything falls in one class.

This matters for post-training because prevalence moves as the model improves. A judge calibrated on a base model that passes 50% of the time will show lower κ when re-checked on a fine-tuned model that passes 90% — even if its error rates are identical. Teams then "recalibrate" a judge that wasn't broken.

Sampling noise compounds it. With 200 human labels and a balanced judge, the 95% interval for κ is [0.71, 0.88] at 50% prevalence and [0.42, 0.74] at 90% (5,000 simulated label sets each). A κ threshold of 0.7 is not a crisp line at n = 200.

The pass rate you report is biased#

The judge's pass rate is $q$, not $\pi$. The bias is $q-\pi =(1-\pi )(1-S_p)-\pi (1-S_e)$: false passes minus false fails.

The lenient judge — the common case for LLM judges, which tend to be generous with fluent, confident answers (Zheng et al., 2023 document verbosity and self-enhancement biases) — overstates a true 50% pass rate as 68.5%, a 18.5-point inflation, while its κ of 0.57 looks only "moderate", not alarming. The strict judge does the opposite and understates good models: at a true 80% it reports 60.6%.

Judges shrink the gap between models#

The most consequential effect for model selection is that judge error compresses differences. Since $q$ is linear in $\pi$,

The factor $J=S_e+S_p-1$ (Youden's index) is always below 1 for an imperfect judge. For the lenient judge $J=0.57$: a true 5-point improvement shows up as 2.85 points. Combined with the eval-size arithmetic in our power analysis, that means a judge-scored eval needs roughly $1/J^2\approx 3.1\times$ as many items to detect the same true improvement.

Correcting the pass rate#

If you have estimated $S_e$ and $S_p$ from a labelled calibration set, you can invert the bias (the Rogan–Gladen estimator, standard in diagnostic testing):

For the lenient judge reporting $q=0.86$, this recovers $\hat{\pi }=0.80$ — the true value. In practice $S_e$ and $S_p$ are themselves estimates, so propagate their uncertainty (a bootstrap over the calibration set is simplest), and recheck them when the output distribution shifts — for example after a large fine-tuning gain.

def corrected_pass_rate(q, se, sp):
    j = se + sp - 1
    if j <= 0.2:
        raise ValueError("judge too close to chance to correct")
    return min(1, max(0, (q + sp - 1) / j))

How many human labels to calibrate a judge#

Sensitivity is estimated only from items humans passed, and specificity only from items humans failed. Their standard errors are

With 200 labels at a 90% pass rate there are only about 20 human fails, so a true specificity of 0.60 is estimated as 0.60 ± 0.21 (95%). That is too wide to correct anything: the Rogan–Gladen denominator $S_e+S_p-1$ could be anywhere from about 0.36 to 0.78. The fix is to stratify the calibration sample — oversample likely failures (low judge scores, known hard slices) until each class has at least 100 human labels — and reweight when reporting.

Pairwise judges and position bias#

Pairwise judges ("which response is better?") have a different dominant error: position bias, a tendency to prefer the first (or second) response regardless of content, documented by Zheng et al. (2023). Model it as the judge picking the truly better response with probability $a$ when it is shown first and $b$ when it is shown second. A single-order evaluation reports a win rate contaminated by the order you happened to use. Evaluating both orders and averaging gives a win-rate estimate whose bias depends on $(a+b)/2$ only, and the disagreement rate between the two orders, $|a-b|$ in expectation, is a free diagnostic: if the judge flips its verdict on more than about 15–20% of pairs when the order is swapped, its preferences are mostly position, not content.

A calibration protocol that holds up#

1. Label on the distribution you'll judge. Sample calibration items from the candidate models, not only from the base model, so prevalence is realistic.
2. Report $S_e$, $S_p$ and κ with intervals. At n = 200, κ alone has an interval roughly ±0.1 wide or wider.
3. Break vague rubrics apart. A "helpfulness 1–5" judge is hard to calibrate; "states the correct refund amount: yes/no" is easy.
4. Correct pass rates with the inverse formula when you report absolute numbers, and remember that gaps between models shrink by $J$.
5. Check position and length bias for pairwise judges by swapping order and comparing lengths of winners (Zheng et al., 2023).

More in LLM-as-a-judge, Evaluating fine-tuned models and the release checklist.

Frequently asked questions#

References#

1. Cohen, J. (1960). A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement 20(1), 37–46. doi:10.1177/001316446002000104. doi.org/10.1177/001316446002000104
2. Feinstein, A. R., Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology 43(6), 543–549. doi.org/10.1016/0895-4356(90)90158-L
3. Rogan, W. J., Gladen, B. (1978). Estimating prevalence from the results of a screening test. American Journal of Epidemiology 107(1), 71–76. doi.org/10.1093/oxfordjournals.aje.a112510
4. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 (NeurIPS 2023). arxiv.org/abs/2306.05685

Related reading