DPO (Direct Preference Optimization), explained

Direct Preference Optimization (DPO) fine-tunes a language model on pairs of responses to the same prompt, one marked better than the other, using a single classification-style loss. It optimizes the same KL-regularized objective as RLHF with PPO, but it never trains a separate reward model and never samples from the policy during training. In the post-training stack it usually sits right after supervised fine-tuning: SFT teaches the format and the task, DPO then shifts probability mass toward the responses people (or a judge model) prefer. Since 2023 it has become the default preference-tuning step in open model recipes, and most of the newer "PO" methods (IPO, KTO, ORPO, SimPO, CPO) are best understood as edits to it.

The problem DPO solves#

The classic RLHF pipeline (Ouyang et al., 2022) has three stages: SFT, then a reward model trained on human comparisons, then PPO to maximize that reward while a KL penalty keeps the policy near the SFT model. The third stage is the expensive and fragile one. PPO keeps four models in play (policy, reference, reward model, value network), generates text inside the training loop, and has a long list of hyperparameters that interact badly.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn (Stanford) posted Direct Preference Optimization: Your Language Model is Secretly a Reward Model in May 2023 (NeurIPS 2023). Their observation: the KL-regularized reward maximization problem has a closed-form optimal policy, so you can rewrite the reward in terms of that policy, substitute it into the preference likelihood, and fit the policy directly by maximum likelihood on comparisons. One model trains, one frozen copy provides the reference, and the loop is ordinary supervised training over a fixed dataset.

The objective, derived#

Start from the RLHF objective

Let $x$ be a prompt, $y$ a response, $r(x,y)$ a reward, ${\pi }_{\text{ref}}$ the SFT model and $\beta >0$ the strength of the KL penalty. RLHF solves

For a fixed reward, the maximizer is a reweighting of the reference policy:

$Z(x)$ sums over every possible response, so it cannot be computed. Take logs and solve for the reward instead:

Plug into Bradley-Terry

Preference data is modeled with Bradley-Terry (Bradley and Terry, 1952): the probability that $y_w$ ("winner") beats $y_l$ ("loser") is $\sigma (r(x,y_w)-r(x,y_l))$, where $\sigma$ is the logistic function. Only reward differences enter, so $\beta \log Z(x)$ cancels. Replacing ${\pi }^{\ast }$ by a trainable ${\pi }_{\theta }$ and maximizing the log-likelihood of the observed comparisons gives the DPO loss:

Every symbol is now computable. $\log {\pi }_{\theta }(y\mid x)$ is the sum of per-token log-probabilities of the response given the prompt (prompt tokens masked out), and the same for the frozen reference. The quantity ${\hat{r}}_{\theta }(x,y)=\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$ is the implicit reward; libraries log it as rewards/chosen and rewards/rejected. Write $h={\hat{r}}_{\theta }(x,y_w)-{\hat{r}}_{\theta }(x,y_l)$ for the implicit reward margin, so the per-pair loss is $-\log \sigma (h)$.

What the gradient does

Each pair pushes the chosen response up and the rejected response down, scaled by $\sigma (-h)$: close to 1 when the model currently ranks the pair the wrong way, close to 0 when it already ranks it correctly by a wide margin. The DPO paper reports that a naive version without this weighting coefficient can make the model degenerate. It also explains a property that surprises people: nothing in the gradient requires $\log {\pi }_{\theta }(y_w\mid x)$ to go up. Only the difference matters.

DPO next to its siblings

Most later methods keep the pairwise structure and change one ingredient: the link function, the reference, or the normalization.

${\Delta }_{\text{ref}}$ is the same difference under the reference. Tang et al. (2024) show that DPO, IPO and SLiC are all instances of one family of convex losses applied to ${\Delta }_{\theta }-{\Delta }_{\text{ref}}$, which is a useful way to read the table: the choice of curve decides how hard the loss pushes on pairs that are already ranked correctly.

A worked example by hand#

Take one pair with $\beta =0.1$. Under the policy, the chosen response has summed log-probability $-42.0$ and the rejected one $-50.0$. Under the reference they were $-43.5$ and $-48.0$.

• Implicit reward of the chosen response: $0.1\times (-42.0+43.5)=0.15$.
• Implicit reward of the rejected response: $0.1\times (-50.0+48.0)=-0.20$.
• Margin $h=0.15-(-0.20)=0.35$.
• Loss $-\log \sigma (0.35)=\log (1+e^{-0.35})=0.533$. At initialization ($h=0$) it was $\log 2=0.693$.
• Gradient weight $\sigma (-0.35)=0.413$: this pair still pulls with 41% of maximum strength.

For calibration: at $h=2$ the loss is 0.127 and the weight 0.119; at $h=5$ both are below 0.007, so a pair with a margin of 5 has effectively stopped contributing. With $\beta =0.1$, a margin of 5 means the log-ratio gap between chosen and rejected has grown by 50 nats. Sequence log-probs are sums over hundreds of tokens, so gaps of that size are reachable, which is why DPO can keep moving long after the loss looks flat.

The algorithm step by step#

Input: SFT model pi_sft, dataset D of (x, y_w, y_l), beta, learning rate
pi_ref  = frozen copy of pi_sft
pi_theta = trainable copy of pi_sft
(optional) for each pair: cache ref_logp_w, ref_logp_l = logp(pi_ref, x, y_w), logp(pi_ref, x, y_l)
for each minibatch B in D (typically 1 to 3 epochs):
    lw = logp(pi_theta, x, y_w)      # sum of token log-probs, prompt masked
    ll = logp(pi_theta, x, y_l)
    h  = beta * ((lw - ref_logp_w) - (ll - ref_logp_l))
    loss = mean(-log_sigmoid(h))
    loss.backward(); optimizer.step()
    log: loss, mean(h > 0), mean beta*(lw - ref_logp_w), mean beta*(ll - ref_logp_l), lw, ll

In practice chosen and rejected sequences are concatenated into one batch so a single forward pass scores both. Nothing is generated. That is the main operational difference from PPO and GRPO: training throughput is close to SFT and the run is deterministic given the data order.

Data it needs

Each row is a prompt plus two complete responses and a label saying which is better. Common sources are human comparisons, an LLM judge ranking several samples (UltraFeedback-style data, which Zephyr used), or a reward model scoring samples from your own SFT model. Dataset sizes in published recipes range from a few thousand pairs for narrow behaviors to a few hundred thousand for general chat: Zephyr-7B used pairs built from UltraFeedback's 64k prompts, and Tulu 3's final mixes held about 271k pairs (8B) and 334k pairs (70B). Quality and coverage matter more than count. Ivison et al. (2024) found data quality moved results more than the choice between DPO and PPO.

The single most useful data property is that the responses come from a model close to the one you are training. Pairs written by a different, much stronger model are "off-policy": the loss can push their probabilities around without the model ever generating anything like them. Tulu 3 reported gains from adding on-policy pairs sampled from its own SFT model, and Llama 3 trained each DPO round mostly on preference data collected with the best models from the previous round. If you can regenerate pairs from the current policy repeatedly, you are doing online or iterative DPO.

Hyperparameters that matter#

β is the inverse temperature of the implicit reward. Small β means a small change in log-ratio is a large change in reward, so the loss is satisfied cheaply and the KL leash is loose. Large β keeps the policy close to the reference. Tune it against a downstream metric and the KL to the reference, not against preference accuracy: in the toy run below, held-out accuracy differs by under two points across a 33x range of β while KL differs by a factor of six.

Learning rate is the knob most often set wrong. DPO gradients are dense (every token of both responses) and the loss has no floor on how far log-probs can fall, so rates that are normal for SFT (1e-5 to 2e-5) frequently wreck a model within a few hundred steps. Llama 3 is a notable exception at 1e-5, but it also added an NLL term and masked formatting tokens to keep training stable. Tune in a coarse grid such as {3e-7, 5e-7, 1e-6} and keep everything else fixed.

Regularizers you can add. An NLL term on the chosen response (coefficient 0.2 in Llama 3, 1.0 in Iterative RPO by Pang et al.) stops chosen log-probs from collapsing. Length-normalizing the log-probs (Tulu 3 used length-normalized DPO with β = 5 and learning rate 5e-7 for 8B) controls verbosity. Label smoothing or the "robust" variant helps when a known fraction of labels is wrong.

Compute and memory#

Per training pair DPO does a forward and backward pass over two sequences on the policy and a forward pass over the same two sequences on the reference. Counting a forward pass as one third of forward plus backward, one pair costs about $2\times (1+\frac{1}{3})\approx 2.7$ times an SFT step on a single response of the same length. If reference log-probs are precomputed in one pass over the data, the per-step cost drops to about 2x SFT plus a one-off inference pass.

Memory for full fine-tuning in mixed precision with Adam is roughly

bytes, with $P$ the parameter count. For a 7B model that is about 112 GB for the policy state plus 14 GB for the reference before activations, so a single 80 GB GPU needs sharding (ZeRO-3 or FSDP), LoRA, or both. With LoRA the reference is free: disable the adapter and the base model is the reference, which is what common implementations do. QLoRA brings 7B to 8B DPO onto a single 24 to 48 GB card at short sequence lengths. Compared with PPO, which adds a reward model, a value model and generation, DPO is typically several times cheaper in wall-clock for the same number of prompts.

A toy run with real numbers#

To show what β does without the noise of a full LLM run, we trained DPO on a log-linear policy over 200 candidate responses for one prompt. Each response has a 16-dimensional feature vector shared through the parameters (so pushing one response moves similar ones, as in a real network), and the true reward is a linear function of those features, standardized to mean 0 and variance 1. The reference assigns random logits. We sampled 3,000 pairs from the reference (2,901 after dropping identical pairs) and labeled them with a Bradley-Terry annotator whose noise caps achievable held-out accuracy at 80.6%. Adam, learning rate 0.02, batch 64, 3,000 steps, five seeds averaged. The script is short numpy plus PyTorch and runs in a few seconds on a laptop CPU.

The reference policy's expected reward is −0.08 and the best single response scores 2.95. Three things stand out. Preference accuracy is nearly flat across β, so it tells you almost nothing about how far the model moved. The chosen log-probabilities fell in every run, by 48 nats at β = 0.03, because the loss only needs the rejected ones to fall faster; the mass went to high-reward responses that never appeared in any pair. And lower β bought more reward at more KL. In this toy the true reward is exactly the direction the preferences reveal, so drift is pure gain. In a real model the KL term is protecting fluency, formatting and every skill the preference data does not mention, which is why practitioners stay near β = 0.1 instead of pushing it toward zero.

For a longer treatment of this experiment, including how near-duplicate pairs make displacement worse, see our β and likelihood displacement study.

Failure modes and how to spot them#

Likelihood displacement. Chosen log-probs fall along with rejected ones. Pal et al. (2024) prove this can happen whenever the relative probability improves, and observe it most on pairs with small edit distance. Razin et al. (2024) show the displaced mass can land on responses with the opposite meaning, for example a refusal dataset that increases unsafe compliance, and trace it to chosen and rejected responses with similar hidden representations. Watch logps/chosen. A steady decline is normal; a decline while your task metric also drops is the warning. Fixes: NLL on chosen (DPO-Positive or the Llama 3 style term), filter near-duplicate pairs, raise β.

Verbosity. Summed log-probs and judge biases both reward longer answers. Park et al. (2024) document significant length exploitation in DPO and propose a length regularizer. Track mean generated length per eval and length-controlled win rates (Dubois et al., 2024). If length climbs steadily while length-controlled scores stall, add length normalization or a length penalty, or rebalance pairs so chosen is not systematically longer.

Overfitting and over-optimization. On deterministic labels the DPO optimum pushes the margin toward infinity regardless of β, which Azar et al. (2023) use to motivate IPO. Symptoms: training accuracy near 100%, eval loss rising after the first epoch, outputs becoming repetitive or oddly terminated. Use one epoch, early stopping on eval loss, or IPO.

Distribution shift. Xu et al. (2024) show theoretically that DPO can find solutions that exploit out-of-distribution responses and report that a well-tuned PPO beat DPO across their testbeds, most clearly on code. The practical remedy is on-policy data, not a different loss.

Formatting collapse. Llama 3's report describes tail repetition and abrupt termination tokens when special formatting tokens contributed to the loss, because the same tokens appear in both responses and receive opposing gradients. Mask chat-template tokens, and check that chosen and rejected were rendered with the same template.

The minimum dashboard: loss, reward accuracy, chosen and rejected implicit rewards, chosen and rejected raw log-probs, an estimate of KL to the reference on generated samples, mean output length, and a downstream eval every few hundred steps.

When to use DPO, and what to use instead#

In POFE, DPO is a standard recipe after SFT, with reference log-probs cached and the full metric set above logged by default, because the most common DPO failure is a run that looks fine on its loss curve.

Minimal implementation#

The core loss is a few lines once you can compute summed response log-probabilities with the prompt masked out.

import torch
import torch.nn.functional as F

def seq_logps(model, input_ids, attention_mask, loss_mask):
    """Sum of log p(token) over response tokens. loss_mask is 1 on response tokens."""
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    labels = input_ids[:, 1:]
    lp = torch.gather(logits.log_softmax(-1), 2, labels.unsqueeze(-1)).squeeze(-1)
    return (lp * loss_mask[:, 1:]).sum(-1)

def dpo_loss(policy, batch, ref_logp_w, ref_logp_l, beta=0.1):
    # batch holds chosen and rejected concatenated along dim 0: [w_1..w_B, l_1..l_B]
    logp = seq_logps(policy, batch["input_ids"], batch["attention_mask"], batch["loss_mask"])
    logp_w, logp_l = logp.chunk(2)
    h = beta * ((logp_w - ref_logp_w) - (logp_l - ref_logp_l))
    loss = -F.logsigmoid(h).mean()
    metrics = {
        "reward_acc": (h > 0).float().mean().item(),
        "reward_chosen": (beta * (logp_w - ref_logp_w)).mean().item(),
        "reward_rejected": (beta * (logp_l - ref_logp_l)).mean().item(),
        "logp_chosen": logp_w.mean().item(),
    }
    return loss, metrics

With Hugging Face TRL the same run is a config object; the dataset needs prompt, chosen and rejected columns.

from trl import DPOConfig, DPOTrainer

args = DPOConfig(output_dir="dpo-out", beta=0.1, learning_rate=5e-7,
                 per_device_train_batch_size=8, gradient_accumulation_steps=8,
                 num_train_epochs=1, warmup_ratio=0.1, bf16=True,
                 precompute_ref_log_probs=True)
trainer = DPOTrainer(model=model, ref_model=None, args=args,
                     train_dataset=pairs, processing_class=tokenizer)
trainer.train()

With ref_model=None the trainer uses the model's initial state as the reference (for a PEFT model, without a second full copy of the weights). The same trainer exposes IPO, hinge and robust losses through loss_type; recent versions also accept a list such as loss_type=["sigmoid", "sft"] with loss_weights to add an NLL term on chosen responses.

Further reading#

The original paper is still the clearest derivation; the rest cover the failure modes and production recipes discussed above.

• Rafailov et al. (2023), Direct Preference Optimization: Your Language Model is Secretly a Reward Model. The original derivation and experiments.
• Azar et al. (2023), A General Theoretical Paradigm to Understand Learning from Human Preferences. Why DPO overfits deterministic preferences; introduces IPO.
• Tang et al. (2024), Generalized Preference Optimization. DPO, IPO and SLiC as one family.
• Razin et al. (2024), Unintentional Unalignment: Likelihood Displacement in DPO, and Pal et al. (2024), Smaug / DPO-Positive.
• Park et al. (2024), Disentangling Length from Quality in DPO.
• Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment?, and Ivison et al. (2024), Unpacking DPO and PPO.
• Tunstall et al. (2023), Zephyr; Lambert et al. (2024), Tulu 3; Llama Team (2024), The Llama 3 Herd of Models. Production recipes.

Frequently asked questions#

References#

1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
2. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arxiv.org/abs/2203.02155
3. Bradley, R. A., Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. doi.org/10.2307/2334029
4. Gheshlaghi Azar, M., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., Munos, R. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arxiv.org/abs/2310.12036
5. Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Ávila Pires, B., Piot, B. (2024). Generalized Preference Optimization: A Unified Approach to Offline Alignment. arxiv.org/abs/2402.05749
6. Razin, N., Malladi, S., Bhaskar, A., Chen, D., Arora, S., Hanin, B. (2024). Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. arxiv.org/abs/2410.08847
7. Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., White, C. (2024). Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive. arxiv.org/abs/2402.13228
8. Park, R., Rafailov, R., Ermon, S., Finn, C. (2024). Disentangling Length from Quality in Direct Preference Optimization. arxiv.org/abs/2403.19159
9. Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T. B. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arxiv.org/abs/2404.04475
10. Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., Wu, Y. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. arxiv.org/abs/2404.10719
11. Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lambert, N., Smith, N. A., Choi, Y., Hajishirzi, H. (2024). Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback. arxiv.org/abs/2406.09279
12. Tunstall, L. et al. (2023). Zephyr: Direct Distillation of LM Alignment. arxiv.org/abs/2310.16944
13. Lambert, N. et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arxiv.org/abs/2411.15124
14. Llama Team, AI @ Meta (Grattafiori, A. et al.) (2024). The Llama 3 Herd of Models. arxiv.org/abs/2407.21783
15. Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., Weston, J. (2024). Iterative Reasoning Preference Optimization. arxiv.org/abs/2404.19733
16. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (DPOTrainer). github.com/huggingface/trl

Related reading