The release-gate checklist for fine-tuned models

Every item on this list exists because a model shipped without it and something went wrong. Copy it into your release template, and make the gate block the deploy rather than merely produce a report.

Eval set#

• ☐ Golden set drawn from production-like inputs, not from training data, and frozen with a version ID.
• ☐ Size chosen from the effect you need to detect (MDE), not from labelling capacity — see sizing table.
• ☐ Slices defined up front (intent, language, customer tier, document type), each with ≥ 50 items.
• ☐ Contamination scan: n-gram (13+) and near-duplicate overlap between every training set and the eval, reported and excluded.
• ☐ Hard cases over-sampled deliberately; every past incident added as an item.

Statistics#

• ☐ Candidate and baseline scored on identical items; paired differences reported.
• ☐ 95% intervals on the primary metric and on every slice.
• ☐ Gate uses the lower bound of the interval, not the point estimate.
• ☐ Sampling variance included if temperature > 0 (multiple seeds).

Judges#

• ☐ Each judge calibrated on ≥ 200 human labels drawn from candidate outputs.
• ☐ Sensitivity, specificity and κ reported with intervals — see what κ hides.
• ☐ Pairwise judges checked for position bias (order swapped) and length bias.
• ☐ Judge model version pinned.

Safety and guardrails#

• ☐ Unsafe-completion rate on an adversarial set at or below the baseline.
• ☐ Benign-refusal rate on a benign set at or below threshold.
• ☐ Memorisation probe: sampled outputs checked for training-set PII or verbatim records.
• ☐ For RL-trained models: reward-hacking review of a sample of high-reward outputs.

Operations#

• ☐ p50/p95 latency and cost per 1k requests measured on the serving stack, not the training box.
• ☐ Chat template and tokenizer at serving identical to training.
• ☐ Rollback path tested (previous adapter/model one command away).
• ☐ Shadow traffic scored by the same evaluators before canary; canary before full rollout.

Template#

gate: support-agent-release-v3
eval_set: golden-support@v7          # frozen
baseline: prod/support-agent@2026-09
primary:
  metric: resolution_rate
  rule: paired_ci95_lower > 0.0      # must beat prod
slices:
  rule: no_slice_ci95_upper < -1.0   # no clear regression
guardrails:
  unsafe_rate: "<= 0.2%"
  benign_refusal: "<= 1.5%"
judges:
  min_kappa: 0.70
  min_labels: 200
hygiene:
  contamination_excluded: true
  max_contamination: "0.5%"
ops:
  p95_latency_ms: 900
  cost_per_1k_requests_usd: 0.40
rollout: [shadow, canary_10, full]

This is the shape POFE gates use. Background reading: Evaluating fine-tuned models and LLM-as-a-judge.

Frequently asked questions#

Related reading