Open the workspace
Glossary

LLM-as-a-judge

LLM-as-a-judge uses a language model to grade outputs. Known biases, how to calibrate against human labels, and which agreement metrics to report.

POFE ResearchPublished 2026-09-05Updated 2026-10-304 min read

LLM-as-a-judge: Using a language model, prompted with a rubric, to score or compare outputs from another model.

Explained#

An LLM judge scores outputs against a rubric (pointwise) or picks the better of two (pairwise). It makes open-ended evaluation scalable, and it is a model with errors and biases of its own. Zheng et al. (2023) found strong judges reach over 80% agreement with human preferences, and documented position, verbosity and self-enhancement biases.

Calibrating a judge

  1. Label 200–500 outputs from the models you'll actually compare.
  2. Report sensitivity and specificity, not only Cohen's κ — κ moves with the pass rate of the outputs even when the judge doesn't change.
  3. Correct reported pass rates: .
  4. For pairwise judges, evaluate both orders and average.

Judge error also compresses gaps between models by the factor , so judge-scored evals need more items. Worked numbers: what Cohen's κ hides.

References#

  1. Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685. arxiv.org/abs/2306.05685
LLM-as-a-judge agreement: what Cohen's κ hidesResearchHow many eval examples do you need to trust a 3-point gain?ResearchThe release-gate checklist for fine-tuned modelsLearnEvaluating fine-tuned models: how to test your evalsGuide

Bring a model and a metric.

Tell us what the model should get better at. A post-training engineer replies within one business day with a plan, a compute estimate and an eval design.