What DPO's β actually controls — and why chosen likelihoods fall

Two things confuse almost everyone the first time they run Direct Preference Optimization. The β hyperparameter seems to do little to the training loss but a lot to the model. And the log-probability of the chosen responses — the ones you told the model you prefer — usually goes down during training. We derive where β comes from, then run a controlled simulation to show both effects with numbers you can reproduce.

Where the DPO loss comes from#

RLHF maximises a reward while staying close to a reference policy (Ouyang et al., 2022):

This objective has a closed-form optimum, ${\pi }^{\ast }(y\mid x)\propto {\pi }_{\mathrm{ref}}(y\mid x)\exp (r(x,y)/\beta )$. Rafailov et al. (2023) invert it: any policy implies a reward, up to a prompt-dependent constant,

Plug that implicit reward into the Bradley–Terry model of pairwise preferences (Bradley & Terry, 1952), $P(y_w\succ y_l)=\sigma (r(x,y_w)-r(x,y_l))$. The intractable $Z(x)$ cancels in the difference, and maximum likelihood on preference pairs becomes

Two consequences matter in practice. First, β is the inverse temperature of the implied reward: a small β means a small change in log-ratio is a large change in reward, so the optimiser can satisfy the preferences with less movement per pair and is penalised less for drifting overall. Second, the loss only constrains the difference of two log-ratios. Nothing in it says the chosen response's probability must rise. Both can fall, as long as the rejected one falls faster.

The gradient makes the weighting explicit. Writing $h$ for the argument of $\sigma$,

Pairs the model already ranks correctly ($h$ large) get little weight. And when $y_w$ and $y_l$ have similar representations, the two gradient terms nearly cancel — the mechanism Razin et al. (2024) identify behind likelihood displacement.

A controlled experiment#

To see these effects without the noise of a full LLM run, we use a log-linear policy over a fixed set of responses, where parameters are shared across responses the way they are in a real network.

• One prompt, $K=400$ candidate responses. Each response $y$ has a unit-norm feature vector $\varphi (y)\in {\mathbb{R}}^{16}$.
• Policy ${\pi }_{\theta }(y)=\mathrm{softmax}({\theta }^{\top }\varphi (y))$. Reference ${\theta }_0\sim \mathcal{N}(0,0.5^{2}I)$.
• True reward $r^{\ast }(y)=w^{\top }\varphi (y)$, scaled to a range of about ±3. Preference labels drawn from Bradley–Terry with $r^{\ast }$, so labels are noisy, as human labels are.
• 2000 training pairs, both responses sampled from ${\pi }_{\mathrm{ref}}$ (offline DPO). 2,000 held-out pairs. Full-batch Adam, learning rate 0.05, 600 steps. 5 seeds per β.

Because the true reward is known, we can measure what DPO cannot see during training: the expected true reward of the learned policy, and how far it moved from the reference.

Result 1: β trades reward against drift, not accuracy#

After 600 steps, mean of 5 seeds. Reference policy expected reward: 0.00. Best single response: 2.04. Accuracy of the true reward on the noisy held-out labels: 68.8%.

Held-out preference accuracy barely moves across a 100× range of β — every setting gets within about two points of the 68.8% ceiling set by label noise. If you tune β by watching preference accuracy, you will conclude it doesn't matter. What changes is how far the policy travels: at β = 1 the KL stays at 0.27 nats and expected reward reaches only 0.53; at β = 0.1 the policy moves 4.84 nats and gets within 0.13 of the best response.

In this toy problem more drift is pure gain, because the true reward is exactly the feature direction the preferences reveal. In a real model the KL term is doing the job the reference was chosen for: keeping fluency, format and everything the preference data says nothing about. That is why production DPO runs typically use β in the 0.05–0.5 range rather than pushing it toward zero, and why KL to the reference is worth logging next to the loss.

Result 2: chosen responses lose probability#

In every run below β = 1, the average log-probability of the chosen responses fell, and fell a lot: -51.3 nats at β = 0.01, -11.9 at β = 0.1. The loss was still doing its job — the rejected responses fell further — but the probability mass went somewhere else.

Where did it go? To responses that appear in neither column. Pairs are sampled from the reference policy, so the chosen response in a pair is usually better than its partner, not the best available. The shared parameters learn the direction of preference, and following that direction moves mass onto the handful of high-reward responses that were rarely sampled. In this setting displacement is benign: expected true reward rose from 0.00 to 1.91.

It is not always benign. Razin et al. (2024) show that in language models the displaced mass can land on responses with the opposite meaning — a refusal training set that increases unsafe compliance — and trace it to preference pairs whose responses have very similar hidden representations.

Result 3: similar pairs displace more#

We tested that mechanism directly. From the same world we built two training sets of 600 pairs each: one where chosen and rejected responses have cosine similarity above 0.6, one where it is below 0.1. Same β (0.1), same steps, 10 seeds.

With similar pairs the chosen log-probability fell by 11.7 nats versus 9.2, and the margin between chosen and rejected was 1.73 versus 4.50 nats. The gradient $\varphi (y_w)-\varphi (y_l)$ is short when the two responses are alike, so each pair moves the "preference direction" less and the shared update pushes both responses together. The policy also ended with lower expected reward (1.64 vs 1.79): near-duplicate pairs are expensive, low-information data.

Practical checklist for DPO runs#

1. Log four numbers, not one: loss, preference accuracy on held-out pairs, KL to the reference, and the mean log-probability of chosen responses. Accuracy alone hides almost everything shown above.
2. Sweep β on your task metric, not on preference accuracy. Start at 0.1; try 0.05 and 0.3.
3. Expect chosen log-probs to fall and investigate only when the task metric falls too, or when samples drift toward content your rejected column was meant to suppress.
4. Audit near-duplicate pairs. Pairs whose responses differ by a few tokens teach little and drive displacement. Embedding similarity between chosen and rejected is a cheap filter; Razin et al. propose a centred-embedding score for exactly this.
5. Use on-policy pairs where possible. Sampling pairs from your current SFT model, rather than from another model, keeps the chosen responses close to where probability mass actually is.

Next: DPO in the glossary, GRPO group size, or the method comparison in SFT vs DPO vs GRPO.

Frequently asked questions#

References#

1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv:2305.18290. arxiv.org/abs/2305.18290
2. Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., Hanin, B. (2024). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. arXiv:2410.08847 (ICLR 2025). arxiv.org/abs/2410.08847
3. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155. arxiv.org/abs/2203.02155
4. Bradley, R. A., Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39(3/4), 324–345. doi.org/10.2307/2334029

Related reading