Reward modeling with Bradley-Terry

A reward model is a network that reads a prompt and a response and returns one number: how much a human (or a stand-in judge) would like that response. The standard way to train one is the Bradley-Terry pairwise loss: show the model two responses to the same prompt, one that annotators preferred and one they rejected, and push the score of the preferred one above the other. In the post-training stack the reward model sits between supervised fine-tuning and reinforcement learning. It is the component that turns a few hundred thousand human judgements into a function you can query millions of times during RLHF with PPO, RLOO, ReMax, best-of-N sampling or rejection sampling fine-tuning. Methods such as DPO fold the same Bradley-Terry model directly into the policy and skip the separate network, which is why understanding this one loss explains most of preference learning.

The problem, and where the model came from#

Humans are poor at assigning absolute scores. Ask ten annotators to rate a summary from 1 to 7 and you get ten private scales, drifting over a working day. Ask the same annotators which of two summaries is better and agreement goes up sharply. Comparisons are cheap, consistent and easy to quality-check. The cost is that you now have ordinal data, and you need a model that converts "A beat B" observations into a scalar score per item.

That model is old. Ralph Bradley and Milton Terry published it in 1952 in Biometrika as a method for analysing paired-comparison taste tests: each item $i$ has a positive "worth" ${\pi }_i$, and the probability that $i$ beats $j$ is ${\pi }_i/({\pi }_i+{\pi }_j)$. Write ${\pi }_i=e^{r_i}$ and the same formula becomes a logistic function of the score difference, which is the form chess ratings (Elo) and modern reward models use.

Deep learning picked it up in 2017, when Christiano, Leike and colleagues trained agents on Atari and MuJoCo from human comparisons of short trajectory clips, fitting a reward network with exactly this loss. Ziegler et al. (2019) moved the setup to language models; Stiennon et al. (2020) used it to train summarisation policies that annotators preferred over the human-written reference summaries; and Ouyang et al. (2022), the InstructGPT paper, made "SFT, then reward model, then PPO" the default recipe for instruction-following assistants. Anthropic's helpful-and-harmless work (Bai et al., 2022) and Llama 2 (Touvron et al., 2023) used the same objective with refinements we cover below.

The objective#

From preferences to a likelihood

Let $x$ be a prompt, $y_w$ the preferred ("winning", chosen) response and $y_l$ the rejected ("losing") one. A reward model $r_{\varphi }(x,y)\in \mathbb{R}$ has parameters $\varphi$. Bradley-Terry says the probability a human prefers $y_w$ is

where $\sigma (z)=1/(1+e^{-z})$ is the logistic sigmoid. Given a dataset $\mathcal{D}=\{(x^{(i)},y_w^{(i)},y_l^{(i)}){\}}_{i=1}^N$, maximum likelihood gives the loss

This is binary cross-entropy on the margin $\Delta r$, with the label always "first one wins". Nothing about the absolute level of $r$ appears in it.

The gradient, and why it is self-limiting

Differentiate with respect to the margin: $\partial \mathcal{L}/\partial \Delta r=-(1-\sigma (\Delta r))$. By the chain rule the parameter gradient is

Two consequences follow. First, the weight $1-\sigma (\Delta r)$ is the model's probability of getting the pair wrong. Pairs the model already ranks confidently contribute almost nothing; pairs it ranks backwards contribute nearly a full unit. Second, the update raises the chosen score and lowers the rejected score by the same amount, so the model learns differences, never levels.

What the scores mean (and what they don't)

Because only $\Delta r$ enters the loss, $r_{\varphi }(x,y)+c(x)$ fits the data exactly as well as $r_{\varphi }(x,y)$ for any per-prompt offset $c(x)$. A reward model's raw output therefore has no meaningful zero, and scores from different prompts are not directly comparable. The scale is pinned: a margin of 1.0 means "73% preference probability" because $\sigma (1)=0.731$. In practice teams normalise before RL. InstructGPT shifted rewards so that labeler demonstrations scored zero on average; most open-source pipelines whiten rewards within a batch. Per-prompt baselines in RLOO and GRPO remove the $c(x)$ ambiguity automatically, which is one reason they are robust to reward models trained this way.

Rankings, margins and ties

K-way rankings. InstructGPT asked labelers to rank between 4 and 9 responses per prompt. A ranking of $K$ responses yields $\left(\genfrac{}{}{0ex}{}{K}{2}\right)$ pairs. Treating those pairs as independent examples overfits, because each response appears in $K-1$ of them; the paper instead put all pairs from one prompt in the same batch and averaged:

The full ranking likelihood is the Plackett-Luce model, which factorises a ranking into successive softmax choices; with $K=2$ it reduces to Bradley-Terry.

Margins. Llama 2 collected a graded preference ("significantly better", "slightly better" and so on) and subtracted a rating-dependent margin $m(r)$ inside the sigmoid:

Larger margins for clear-cut pairs force wider separation where annotators were confident, and the authors report it helped most on those pairs.

Noise. Christiano et al. assumed annotators answer uniformly at random some fraction of the time (they used 10%), giving $P=0.9\sigma (\Delta r)+0.05$. This bounds the loss on any single pair and limits how far a mislabelled example can drag the model. Label smoothing achieves a similar effect.

Relation to its siblings

The same likelihood shows up across preference learning, with a different choice of what plays the role of $r$:

DPO's derivation starts from the KL-regularised RL objective, solves for the optimal policy, inverts it to express $r$ through $\pi$, and substitutes into ${\mathcal{L}}_{\text{BT}}$. The per-prompt constant $c(x)$ we said was unidentifiable is exactly the log partition function that cancels in that substitution. If you understand why a reward model only learns differences, you understand why DPO works without ever computing $Z(x)$.

Worked example by hand#

Take one pair where the model currently scores the chosen response at $r_w=1.2$ and the rejected one at $r_l=0.4$. The margin is 0.8, so the model's preference probability is $\sigma (0.8)=0.690$ and the loss is $-\log 0.690=0.371$. The gradient with respect to $r_w$ is $-(1-0.690)=-0.310$: gradient descent raises $r_w$ by 0.31 times the learning rate and lowers $r_l$ by the same amount.

Flip the scores ($r_w=0.4,r_l=1.2$) and the model is wrong: $p=0.310$, loss 1.171, gradient magnitude 0.690, more than twice as large. Now take a pair the model ranks confidently, $r_w=3.0,r_l=-1.0$: $p=0.982$, loss 0.018, gradient 0.018. That pair is effectively finished. A batch's loss is dominated by its misranked and borderline pairs, which is why reward-model training curves drop fast and then flatten: the easy pairs saturate early and the rest are often genuinely ambiguous.

The algorithm and the data#

Architecture

Start from a pretrained or SFT checkpoint, drop the language-model head and add a linear layer from the hidden size to one output. Run the concatenated prompt and response through the backbone and read the scalar at the final token (the last non-padding position). The SFT checkpoint is the usual initialisation because it already models the response distribution the RM will be asked to score. InstructGPT used a 6B reward model for a 175B policy, reporting that 175B reward models were less stable to train; later work has generally found that larger reward models are more accurate and more robust to optimisation pressure, so most teams now use an RM at least as large as the policy when they can afford it.

Data format and size

One record is a prompt plus two completions with a label. The completions should come from the distribution you will later optimise: samples from your SFT model, ideally at several temperatures and from a few checkpoints, so the RM sees the kind of mistakes the policy will make. Pairs of a strong model's output against a weak model's output teach the RM to detect "which model wrote this", which is easy and useless.

{"prompt": "Explain why the sky is blue in two sentences.",
 "chosen":   "Sunlight scatters off air molecules, and shorter blue wavelengths scatter far more ...",
 "rejected": "The sky is blue because it reflects the ocean ...",
 "margin": 2}      # optional graded strength, for a margin loss

Sizes that work in practice: 5k to 20k pairs give a usable narrow-domain RM (one product surface, one style guide); 50k to 300k pairs is typical for a general assistant RM; frontier labs report datasets in the low millions of comparisons accumulated over many rounds. Inter-annotator agreement sets your ceiling. If two humans agree on 70% of pairs, a model scoring 70% held-out accuracy has learned about as much as the labels contain.

The training loop

init RM from SFT checkpoint; replace LM head with Linear(d_model, 1)
for epoch in 1..E:                      # E is usually 1
    for batch of (x, y_w, y_l) in D:    # all pairs from one prompt in one batch if ranked
        r_w = RM(x ⊕ y_w)[last_token]
        r_l = RM(x ⊕ y_l)[last_token]
        loss = -logsigmoid(r_w - r_l - margin).mean()
        loss.backward(); optimizer.step()
    eval: held-out pairwise accuracy, mean margin, calibration
after training: compute mean reward on a reference set; store offset for normalisation

Chosen and rejected are usually concatenated along the batch dimension so one forward pass scores both. One epoch is the common default: Llama 2 reported that training longer led to overfitting, InstructGPT found that treating every pair from a ranking as an independent example overfit within a single epoch, and a second pass over the same pairs mostly sharpens confidence on examples the model already ranks correctly.

Hyperparameters that matter#

Compute and memory#

Training a reward model costs about the same as an SFT run of the same size on the same number of tokens, times two: each example has two responses. The scalar head adds nothing measurable. For full fine-tuning with Adam in mixed precision, the usual rule of thumb is about 16 bytes per parameter for weights, gradients and optimiser state (2 + 2 + 12), so a 7B reward model needs roughly 112 GB before activations. That is two 80 GB GPUs with ZeRO-style sharding, or one GPU with LoRA (the frozen bf16 base is 14 GB) and gradient checkpointing.

The bigger cost is downstream. During PPO the reward model is one of four networks in memory (policy, value model, reference policy, reward model) and it scores every rollout. A 7B RM scoring 512 rollouts of 1,000 tokens is a forward pass over half a million tokens per RL step. That ongoing inference cost is what DPO avoids and what makes RM size a real trade-off against policy size.

A toy experiment: how much data, how much noise#

To see the learning dynamics without a GPU, we simulated the whole pipeline with a linear reward model. Each "response" is a 16-dimensional feature vector; a hidden true reward $r^{\ast }(y)=w^{\ast \top }y$ with $\Vert w^{\ast }\Vert =2$ generates labels by sampling from the Bradley-Terry model itself, so even the true reward does not predict every label. We fitted $w$ by gradient descent on ${\mathcal{L}}_{\text{BT}}$ (300 steps, learning rate 0.5, weight decay 1e-3, seed 0), with and without 20% of labels flipped at random, and scored two numbers on 20,000 fresh pairs: agreement with the true ranking, and accuracy against freshly sampled noisy labels, which is what a held-out human test set measures.

Three things show up. The held-out label accuracy has a hard ceiling: the true reward itself scores 0.828 against sampled labels, and by 1,000 clean pairs the model is within 0.3 points of it while its agreement with the true ranking keeps climbing from 0.968 to 0.992. Accuracy against noisy labels badly understates how much a reward model has improved, which is why RM leaderboards built on "chosen vs rejected" accuracy saturate in the 70s and 80s. Second, flipping 20% of labels costs between five and twenty times the data: the noisy run at 5,000 pairs (0.955) is still behind the clean run at 1,000 (0.968), and only passes it somewhere before 20,000. Third, label noise shrinks the learned scale. The clean fit recovers $\Vert w\Vert \approx 1.98$; the noisy fit converges to about 0.71. The direction is right (cosine 0.998 with $w^{\ast }$) but every margin is compressed by a factor of nearly three, so a KL coefficient tuned against one reward model is wrong for another trained on noisier data.

Overoptimisation in the same toy

We then changed one thing: annotators over-reward one feature (call it length) with weight 1.2, while the true utility rewards it only slightly and penalises extremes, $r^{\ast }(y)=q(y)+0.3L-0.3L^2$. We trained a reward model on 5,000 annotator pairs and ran best-of-N selection against it, which has a closed-form KL from the base distribution of $\log N-(N-1)/N$. The RM learned a length weight of 1.20, faithfully copying the annotators' bias, and its held-out accuracy against the true utility was 0.675 (ceiling 0.742).

The proxy score climbs monotonically; true reward peaks around N = 32 (2.5 nats) and then falls while the selected responses keep getting longer. This is the shape Gao, Schulman and Hilton (2022) measured at scale with a large "gold" reward model standing in for humans: for best-of-N they fitted gold reward as $d(\alpha -\beta d)$ with $d=\sqrt{\mathrm{KL}}$, a rise followed by decline, and found the peak moves outward as the proxy RM gets larger or is trained on more data.

Failure modes and how to detect them#

Spurious features. Length is the classic one; formatting (bullet lists, headers), hedging phrases and confident tone are others. Detect it by correlating RM score with response length on a fixed set of policy samples, and by building contrast pairs that differ only in that feature (same content, padded or bulleted). If the RM prefers the padded version more than 60% of the time, the policy will learn to pad.

Distribution shift. The RM was trained on SFT samples; after a few hundred RL steps the policy produces text it never saw. Accuracy on an old held-out set says nothing about accuracy on the new distribution. Label a small fresh set (a few hundred pairs) from current policy samples every round and track accuracy there.

Overfitting. Training loss keeps falling in epoch 2 while held-out accuracy stalls and the mean absolute margin on held-out pairs balloons. The RM becomes overconfident, and RL exploits overconfident regions first.

Reward hacking during RL. RM score rises steadily while human or LLM-judge win rate plateaus or drops, KL from the reference grows fast, and samples converge on a template. Track true win rate against KL, not against steps; the curve should look like the toy chart above, and you want to stop before the peak.

Miscalibration. Bin held-out pairs by predicted probability and compare to empirical agreement. A well-trained BT model is reasonably calibrated; a margin-loss model is systematically overconfident by construction, which matters if you use the probabilities (for example to weight pairs or filter data).

Collapsed or drifting scale. Mean reward on a fixed reference set drifts between checkpoints. Harmless for ranking, harmful when the downstream RL uses absolute rewards with a fixed KL coefficient. Normalise against a reference set and record the offset alongside the checkpoint.

Evaluating a reward model#

Held-out pairwise accuracy is the first number, but it measures agreement with noisy labels on the training distribution. RewardBench (Lambert et al., 2024) tests reward models on curated chosen-rejected pairs across chat, hard chat, safety and reasoning, and is a useful sanity check against public baselines. The test that matters most is downstream: run best-of-N with $N$ of 4 to 64 on your own prompts and measure win rate with humans or a strong judge. A reward model that ranks well but whose best-of-16 output does not beat best-of-1 will not help RL either.

When to use it, and what to use instead#

Minimal implementation in PyTorch#

This uses Hugging Face Transformers for the backbone. TRL's RewardTrainer wraps the same loss with batching and logging; writing it by hand shows that there is nothing more to it.

import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForSequenceClassification

name = "path/to/sft-checkpoint"
tok = AutoTokenizer.from_pretrained(name)
tok.padding_side = "right"
rm = AutoModelForSequenceClassification.from_pretrained(name, num_labels=1, torch_dtype=torch.bfloat16)
rm.config.pad_token_id = tok.pad_token_id      # head reads the last non-pad token
opt = torch.optim.AdamW(rm.parameters(), lr=5e-6, weight_decay=0.0)

def score(prompts, responses):
    enc = tok([p + r for p, r in zip(prompts, responses)], return_tensors="pt",
              padding=True, truncation=True, max_length=2048).to(rm.device)
    return rm(**enc).logits.squeeze(-1).float()      # shape [B]

def step(batch, margin=None):
    B = len(batch["prompt"])
    r = score(batch["prompt"] * 2, batch["chosen"] + batch["rejected"])   # one pass, 2B rows
    r_w, r_l = r[:B], r[B:]
    delta = r_w - r_l - (margin if margin is not None else 0.0)
    loss = -F.logsigmoid(delta).mean()
    loss.backward(); opt.step(); opt.zero_grad()
    return loss.item(), (r_w > r_l).float().mean().item(), delta.abs().mean().item()

Log all three returned values: loss, batch accuracy and mean absolute margin. A run that is learning shows loss falling from 0.693, accuracy rising and margins growing slowly; a run that is overfitting shows margins growing quickly while held-out accuracy stops moving. In POFE a reward-model run is a recipe step whose output is versioned with its normalisation offset, so the downstream RL stage always reads matching numbers.

Further reading#

Bradley and Terry (1952) for the original model. Christiano et al. (2017) for the first deep RL use. Stiennon et al. (2020) and Ouyang et al. (2022) for the language-model recipe, including K-way rankings and reward normalisation. Touvron et al. (2023) for margin losses and separate helpfulness and safety reward models. Gao, Schulman and Hilton (2022) for overoptimisation scaling laws. Lambert et al. (2024) for RewardBench. Rafailov et al. (2023) for how DPO folds this loss into the policy.

Frequently asked questions#

References#

1. Ralph Allan Bradley, Milton E. Terry (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. doi.org/10.2307/2334029
2. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, Dario Amodei (2017). Deep reinforcement learning from human preferences. arxiv.org/abs/1706.03741
3. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, et al. (2019). Fine-Tuning Language Models from Human Preferences. arxiv.org/abs/1909.08593
4. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, et al. (2020). Learning to summarize from human feedback. arxiv.org/abs/2009.01325
5. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. (2022). Training language models to follow instructions with human feedback. arxiv.org/abs/2203.02155
6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arxiv.org/abs/2204.05862
7. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arxiv.org/abs/2307.09288
8. Leo Gao, John Schulman, Jacob Hilton (2022). Scaling Laws for Reward Model Overoptimization. arxiv.org/abs/2210.10760
9. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. arxiv.org/abs/2403.13787
10. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290

Related reading