Evaluating fine-tuned models: how to test your evals

Most fine-tuning projects that fail don't fail in training. They fail in evaluation: a model ships because a number went up, and the number was measuring the wrong thing, or measuring it badly. This guide is about evaluating fine-tuned LLMs well — and about the step most teams skip, testing the evals themselves.

Why evals fail

Four failure modes cause most bad releases:

• Unrepresentative test sets. The eval was assembled from easy, clean examples while production traffic is messy, multilingual and adversarial.
• Contamination. Eval items, or near-copies, ended up in training data. Scores rise; real performance doesn't.
• Uncalibrated judges. An LLM-as-judge rates outputs, but nobody checked whether it agrees with human experts — and it often prefers longer, more confident answers.
• Averages that hide regressions. The overall score improves while a critical slice — refunds, a minority language, rare codes — gets worse.

Each one is detectable with a specific test. Treat your eval suite as code: it needs its own tests.

Build a golden set that looks like production

Start from real traffic, not from your training data. Sample production inputs (or realistic proxies), stratify by the slices that matter — intent, customer segment, language, document type, difficulty — and have experts write or approve the expected outputs.

• Size: 300–1,000 items is enough for most single-task evals. You need enough per slice to detect meaningful differences; 50+ per slice is a sensible floor.
• Hard cases: deliberately over-sample known failure modes and edge cases. An eval where every model scores 98% teaches you nothing.
• Versioning: freeze each version and record who approved it. Scores are only comparable on the same version.
• Separation: keep the golden set in a location training pipelines can't read from.

Choose metrics you can defend

Prefer metrics that can be computed deterministically, and use judges only where you must.

Always track guardrail metrics alongside the target: refusal rate on benign requests, unsafe completions on adversarial ones, output length, latency and cost per request. A model that improves accuracy by being twice as long and twice as slow may not be an improvement.

Calibrating LLM-as-judge

LLM judges make open-ended evaluation scalable, but an uncalibrated judge is a random number generator with good grammar. Calibrate before you trust it.

1. Collect human labels on 200–500 outputs, sampled across models and slices, using the same rubric the judge will use. Have at least two experts label a subset to measure human–human agreement.
2. Measure judge–human agreement per rubric criterion with Cohen's κ (or Spearman correlation for scales). κ above 0.7 is good; below 0.5 means the judge isn't measuring what the humans are.
3. Fix disagreements. Tighten the rubric, add examples to the judge prompt, split vague criteria ("helpfulness") into specific ones ("answers the question asked", "states the correct refund amount").
4. Check for known biases: position bias in pairwise comparisons (swap order and average), length bias, and self-preference when the judge is from the same family as a candidate.
5. Re-calibrate when the rubric, judge model or production distribution changes.

The judge is a model. Evaluate it like one.

Contamination and leakage checks

Contamination is the most common reason a fine-tuned model looks better offline than online. Check for it explicitly between every training set and every eval set.

• Exact and n-gram overlap. Flag eval items that share long n-grams (13+ tokens is a common threshold) with any training example.
• Near-duplicate detection. MinHash or embedding similarity catches paraphrases and templated variants that n-grams miss.
• Synthetic data lineage. If a teacher model generated training data from prompts that overlap the eval, that is leakage even if no text matches.
• Temporal splits. Where possible, train on data before a date and evaluate on data after it.

Report the overlap rate per eval set and exclude contaminated items from release decisions.

Is the improvement real? Statistics that matter

With a few hundred eval items, a two-point difference may be noise. Before claiming a win:

• Bootstrap confidence intervals. Resample the eval set with replacement 1,000+ times and report the 95% interval of the score difference between models.
• Paired comparisons. Compare models on the same items; paired tests are far more sensitive than comparing two independent averages.
• Sampling variance. If you sample at temperature above zero, run several seeds and include that variance.
• Slice-level results. Report every slice with its own interval, and require no significant regression on any critical slice.

Evaluating multi-turn conversations and agents

Single-turn evals miss the failures that matter most for assistants and agents: losing track of context, repeating questions, taking an irreversible action too early, or recovering badly from a tool error. Evaluating these needs a different setup.

Simulated users

For conversational systems, script a set of user personas and goals — a customer who wants a refund they're not entitled to, an impatient user who answers in fragments, a user who switches language mid-conversation — and have a separate model play each persona against the candidate. Score whole conversations on goal completion, policy compliance and turns to resolution. Keep persona prompts and seeds fixed so results are comparable between runs.

Sandboxed task environments

For agents that call tools, build a sandbox that mirrors production APIs with realistic data and deterministic state. Each eval task has a start state, an instruction and a checker that inspects the end state: was the ticket closed with the right tag, does the database contain the correct row, was the email drafted but not sent. End-state checking is far more reliable than judging the transcript.

What to report

• Task success rate, with confidence intervals, per task family.
• Unsafe or irreversible actions, counted separately and held to a near-zero threshold.
• Efficiency: steps, tool calls, tokens and wall-clock time per successful task.
• Recovery: success rate on tasks where a tool is made to fail deliberately.

Agent evals are noisier than single-turn ones because small differences early in an episode compound. Run each task several times and report the variance, and be wary of declaring a winner from a single pass.

From eval to release gate

Evaluation becomes useful when it decides what ships. Define a release gate before training:

gate release-v2:
  primary:   resolution_rate  >= prod + 2.0 pts  (95% CI lower bound > 0)
  slices:    no slice regresses beyond CI
  guardrail: unsafe_rate <= 0.2%  ·  benign_refusal <= 1.5%
  ops:       p95 latency <= 900 ms  ·  cost/1k req <= $0.40
  hygiene:   contamination <= 0.5% (excluded items)  ·  judge κ >= 0.7

Then ship gradually: shadow traffic first, scored by the same evaluators, then a small canary, then full rollout. Feed production failures back into the golden set — every incident should become an eval item.

POFE builds all of this in: calibrated judges, contamination scans, bootstrap intervals and release gates are part of every run. See it on the Evaluate step of the interactive demo, or read SFT vs DPO vs GRPO to choose the training method your eval will judge.

Frequently asked questions