SimPO (Simple Preference Optimization), explained

SimPO (Simple Preference Optimization) is a reference-free variant of DPO that scores each response by its average per-token log-probability under the policy, instead of its summed log-ratio against a frozen reference model, and asks the chosen response to beat the rejected one by a fixed margin $\gamma$. Two changes, and they buy two things: the reward the loss optimizes is the same quantity the model uses when it generates, and there is no reference model to store or run. SimPO sits where DPO does in the post-training stack, after SFT, trained offline on preference pairs, and it is a strong default when memory is tight or verbosity is a concern.

Where SimPO came from#

Yu Meng, Mengzhou Xia and Danqi Chen (University of Virginia and Princeton) posted SimPO: Simple Preference Optimization with a Reference-Free Reward in May 2024. They start from a mismatch in DPO. DPO's implicit reward is $\beta \log \frac{{\pi }_{\theta }(y\mid x)}{{\pi }_{\text{ref}}(y\mid x)}$, but at inference time nothing involves a reference model: decoding favors sequences with high average log-likelihood under ${\pi }_{\theta }$. So a DPO-trained model can rank a pair "correctly" by its reward while ranking it the other way by the likelihood that actually drives generation. Measured on their UltraFeedback training set with a DPO-trained model, only about half of the triples where the DPO reward favored the winner also had the winner ahead in average log-likelihood. DPO's reward, in other words, was close to a coin flip as a predictor of what the model would prefer to generate.

The obvious fix, using $\log {\pi }_{\theta }(y\mid x)$ itself as the reward, fails in a different way: summed log-probability is lower for longer sequences, so the loss can be satisfied by shifting length rather than quality. Normalizing by length fixes that. A margin term, borrowed from the Bradley-Terry-with-margin idea, keeps the separation meaningful once the reference is gone.

The objective#

SimPO's reward is the length-normalized log-likelihood, scaled by $\beta$:

$|y|$ is the number of response tokens. Plugging this into a Bradley-Terry likelihood with a target margin $\gamma >0$ gives

Side by side with DPO:

Three differences: no ${\pi }_{\text{ref}}$, division by length, and the $-\gamma$ inside the sigmoid. Because rewards are now per-token averages, the useful range of $\beta$ is much larger than in DPO (the paper uses 2 to 2.5, and up to 10 in one setting), and $\gamma$ is easiest to reason about as the ratio $\gamma /\beta$: the gap in average log-probability, in nats per token, that the chosen response must exceed before the loss stops pushing hard.

The gradient

Writing $z=\frac{\beta }{|y_w|}\log {\pi }_{\theta }(y_w)-\frac{\beta }{|y_l|}\log {\pi }_{\theta }(y_l)-\gamma$,

Each token's gradient is weighted by $1/|y|$, so a long response does not get more total push than a short one simply by having more tokens. In DPO's gradient the per-token weights are all 1, and the response with more tokens dominates. The $\sigma (-z)$ weight behaves like DPO's, except that it stays near 1 until the normalized margin exceeds $\gamma$, not merely zero.

Relation to its siblings

SimPO is one point in a small design space. CPO also drops the reference but uses summed log-probs and adds an NLL term. ORPO length-normalizes, uses odds instead of probabilities, and adds NLL so it can start from a base model. Length-normalized DPO keeps the reference and divides each log-ratio by its length; Tulu 3 compared it with SimPO and DPO and found it worked best in their setting, using $\beta =5$ and learning rate 5e-7 at 8B. R-DPO keeps DPO and adds an explicit length penalty. And SLiC-HF and RRHF, ranking losses with an NLL term, were among the baselines SimPO was tuned against.

Worked example#

A chosen response of 120 tokens with summed log-probability $-60$ (average $-0.5$ per token), a rejected response of 50 tokens with summed log-probability $-40$ (average $-0.8$). Take $\beta =2$, $\gamma =1$.

• $r_w=2\times (-60/120)=-1.0$, $r_l=2\times (-40/50)=-1.6$. Normalized margin $0.6$.
• Inside the sigmoid: $0.6-1=-0.4$. Loss $-\log \sigma (-0.4)=0.913$. Gradient weight $\sigma (0.4)=0.599$.
• Per token the model already prefers the chosen response, but not by the required $\gamma /\beta =0.5$ nats per token. It keeps pushing.
• A summed-log-probability reward sees this pair the other way round: $-60$ against $-40$, a 20-nat preference for the rejected response, created entirely by its being shorter. A reference-free loss on summed log-probs would spend its gradient on length.

What the paper reports#

The authors tuned every baseline (DPO, IPO, KTO, ORPO, CPO, R-DPO, RRHF, SLiC-HF) per setting, with learning rates searched over {3e-7, 5e-7, 6e-7, 1e-6}, batch size 128 and one epoch fixed after preliminary search. SimPO outperformed DPO by up to 6.4 points on AlpacaEval 2 and up to 7.5 points on Arena-Hard. On Llama-3-8B-Instruct it reached 44.7% length-controlled win rate on AlpacaEval 2 and 33.8% on Arena-Hard, against DPO's 40.3% and 32.6%. Re-ranking the on-policy training data with a stronger reward model (ArmoRM instead of PairRM) lifted those to 53.7% and 36.5%, a bigger jump than the algorithm change itself, which is worth remembering. Their strongest model, Gemma-2-9B-it-SimPO, reached 72.4% length-controlled AlpacaEval 2 and 59.1% on Arena-Hard.

The ablations support both design choices. In the Mistral-Instruct setting, removing length normalization dropped length-controlled AlpacaEval 2 from 32.1% to 19.1%, below DPO's 26.8%; setting $\gamma =0$ dropped it to 30.9%. Without normalization the Spearman correlation between average log-probability and response length rose to 0.82, against 0.34 for SimPO and 0.59 for DPO. Against a vanilla DPO implementation SimPO cut run time by roughly 20% and GPU memory by about 10% on 8 H100s, by removing the reference forward passes. The authors note in a footnote that DPO can match SimPO's memory if the reference passes are separated from training, for example by precomputing them.

The paper is also candid about a limit: with a strong starting model and high-quality on-policy data (their Llama-3-Instruct setting with ArmoRM labels), differences between algorithms shrink. There DPO matched SimPO's raw win rate, and SimPO's lead in length-controlled scores came from producing shorter outputs.

A toy run: length and quality#

To see the length mechanism directly, we used a unigram language model with an end-of-sequence token: 10 content tokens (5 "good", 5 "plain") plus EOS, starting from a reference with expected length 25 tokens and 43% good tokens. We sampled 4,000 pairs from the reference and labeled them with a Bradley-Terry annotator that rewards the share of good tokens and also, like many human and LLM judges, longer answers (score $=6q+0.02\ell$). In the resulting data chosen responses average 30.5 tokens and rejected 20.1, and chosen is longer in 61% of pairs. We trained three objectives with Adam (learning rate 0.01, batch 64; one epoch is about 63 steps): DPO ($\beta =0.1$), SimPO ($\beta =2,\gamma =1$) and SimPO without length normalization (summed log-probs, $\beta =0.1$, $\gamma =0$).

The quality column is the clearest result. Without normalization the objective learned almost nothing about which tokens are good (0.43 to 0.45): the length difference between chosen and rejected dominates the summed log-prob difference, so the gradient goes into making sequences longer. DPO learned a little. SimPO learned the quality signal quickly and kept learning. That matches the paper's ablation, where removing normalization was the most damaging change.

The length column is less flattering, and we report it as measured. All three objectives lengthened outputs, SimPO by about as much as DPO at this level of annotator length bias. In a unigram model the EOS probability is a single parameter shared by every position, and the data itself says longer is preferred, so any objective that fits the labels learns some of that. We swept the annotator's length bias to check the trend:

With little or no length bias in the labels, SimPO lengthened outputs least. As the bias grows, its advantage on length disappears and at the strongest bias it lengthens most by three epochs, while its quality advantage stays large throughout. The practical reading: length normalization removes the length bias that comes from the loss; it cannot remove a length bias that is in the labels. That still has to be handled in the data, for example by balancing pairs on length or using length-controlled judging.

The algorithm and data#

Input: SFT model pi_theta, pairs D = {(x, y_w, y_l)}, beta, gamma
for each minibatch (1 epoch is typical):
    avg_w = sum_logp(pi_theta, x, y_w) / len(y_w)     # mean token log-prob, prompt masked
    avg_l = sum_logp(pi_theta, x, y_l) / len(y_l)
    z     = beta * (avg_w - avg_l) - gamma
    loss  = mean(-log_sigmoid(z))
    step optimizer
    log: loss, mean(beta*avg_w), mean(beta*avg_l), reward accuracy, generated length

Data is standard preference pairs. In the paper's Instruct settings, which produced its strongest models, the pairs were close to on-policy: for each UltraFeedback prompt, sample five responses from the starting model at temperature 0.8, score them with a reward model, and take the highest as chosen and the lowest as rejected. (The Base settings used the off-the-shelf binarized UltraFeedback pairs.) This matters more for SimPO than for DPO, because without a reference model nothing anchors the policy to responses outside the data; on-policy pairs keep the data where the policy's mass actually is. With off-policy data from much stronger models, expect less benefit.

What to log, and what a healthy run looks like

SimPO's metrics start from different places than DPO's, which trips people up when they reuse a DPO dashboard. In DPO every implicit reward is exactly zero at step zero, so the loss starts at $\log 2=0.693$ and reward accuracy starts undefined or near 50%. In SimPO the rewards at step zero are $\beta$ times the SFT model's average log-probabilities, so reward accuracy starts at whatever fraction of pairs the SFT model already ranks correctly per token, and the initial loss depends on $\gamma$: with an initial normalized margin near zero and $\gamma =1$, it starts near $-\log \sigma (-1)=1.31$, not 0.69. A healthy run shows the normalized margin climbing past $\gamma$ on most pairs within the epoch, chosen average log-prob roughly flat, rejected average log-prob falling, and generated length stable. Because there is no reference model in the loop, also keep a frozen copy of the starting model for evaluation only and compute KL on generated samples every few hundred steps. It costs a little extra inference, but it is the only direct view you have of how far the policy has moved, and SimPO gives you no other brake.

Hyperparameters#

$\beta$. Start at 2.0 to 2.5. Larger $\beta$ sharpens the sigmoid on normalized rewards; the paper found $\beta =10$ better in one Llama-3-Instruct setting. Its effect on drift is the opposite of DPO's intuition, because there is no reference term for $\beta$ to scale: it is a temperature on per-token rewards.

$\gamma$. Search $\gamma /\beta$ between roughly 0.1 and 0.8 (the paper's search covered $\gamma$ from 0.3 to 1.6). The paper's analysis: reward accuracy on held-out pairs rises steadily with $\gamma$, but AlpacaEval 2 win rate rises and then falls, and large $\gamma$ flattens the reward distributions and lowers the average log-likelihood of winning responses, eventually degenerating the model.

Learning rate. The most sensitive knob, and the paper says so explicitly: the optimal rate differed by method and setting and greatly influenced benchmark results. Search a small grid in the 3e-7 to 1e-6 range for full fine-tuning at 7B to 9B. Batch size 128 and one epoch were best across methods in their preliminary search.

Compute and memory#

SimPO runs one model. Per pair: forward and backward through the policy on chosen and rejected, about 2x an SFT step on a single response, against about 2.7x for DPO without precomputed reference log-probs. Memory is that of SFT on two sequences per example: roughly 16 bytes per parameter for full fine-tuning with Adam in mixed precision, plus activations; no frozen copy. The paper measured about 20% less run time and 10% less peak memory than a vanilla DPO implementation. If you already precompute DPO's reference log-probs, the speed gap mostly closes; the real differences are then the reward definition and the margin, not cost. With LoRA the reference is free for DPO too, so SimPO's cost advantage matters most for full fine-tuning.

Failure modes#

No anchor. Nothing in the loss penalizes moving away from the SFT model in directions the pairs do not test. The usual symptoms are degraded general knowledge or math after many steps, or formatting drift. Evaluate broad benchmarks, not only the chat metric, and prefer one epoch. CPO-style NLL on chosen responses (the TRL implementation exposes this as a weight) is a common patch.

Learning rate too high. Average log-probs of both chosen and rejected fall quickly, outputs become repetitive or truncated. Lower the rate before touching $\beta$ or $\gamma$.

$\gamma$ too large. Reward accuracy looks excellent while win rate falls and chosen likelihood drops. Track average log-prob of chosen responses: a steady decline is the warning.

Length bias from the data. As the toy shows, normalization fixes the loss's own length incentive but not a judge that prefers long answers. Monitor generated length and length-controlled metrics, and balance pairs if chosen is systematically longer.

Off-policy data. Pairs written by another model can be separated by shifting probability onto text the policy would never produce. The paper's strongest results used pairs sampled from the starting model; replicate that if you can.

When to use SimPO#

In POFE, SimPO is a recipe option alongside DPO on the same pairs, which makes the two cheap to compare with identical data and evaluation.

Minimal implementation#

import torch
import torch.nn.functional as F

def simpo_loss(logp_w, len_w, logp_l, len_l, beta=2.0, gamma=1.0):
    """logp_*: summed response log-probs [B]; len_*: response token counts [B]."""
    r_w = beta * logp_w / len_w
    r_l = beta * logp_l / len_l
    z = r_w - r_l - gamma
    loss = -F.logsigmoid(z).mean()
    return loss, {"reward_chosen": r_w.mean().item(), "reward_rejected": r_l.mean().item(),
                  "reward_acc": (r_w > r_l).float().mean().item(),
                  "margin_minus_gamma": z.mean().item()}

TRL implements SimPO through the CPO trainer (under trl.experimental in recent versions). Set cpo_alpha=0 for pure SimPO; a positive value adds CPO's NLL term on chosen responses. simpo_gamma is $\gamma$ itself.

from trl.experimental.cpo import CPOConfig, CPOTrainer

args = CPOConfig(output_dir="simpo-out", loss_type="simpo", beta=2.0, simpo_gamma=1.0,
                 cpo_alpha=0.0, learning_rate=5e-7, num_train_epochs=1,
                 per_device_train_batch_size=4, gradient_accumulation_steps=32, bf16=True)
trainer = CPOTrainer(model=sft_model, args=args, train_dataset=pairs,
                     processing_class=tokenizer)
trainer.train()

Further reading#

The SimPO paper's appendix tables list per-setting hyperparameters and the tuned baselines referenced above.

• Meng, Xia and Chen (2024), SimPO: Simple Preference Optimization with a Reference-Free Reward. Method, tuned baselines, ablations and hyperparameter tables.
• Rafailov et al. (2023), Direct Preference Optimization. The reward SimPO replaces.
• Park et al. (2024), Disentangling Length from Quality in DPO. Length exploitation in DPO and a regularizer for it.
• Dubois et al. (2024), Length-Controlled AlpacaEval. The length-controlled metric SimPO is evaluated with.
• Lambert et al. (2024), Tulu 3. A large-scale comparison where length-normalized DPO beat SimPO.
• Xu et al. (2024), CPO, and Hong et al. (2024), ORPO. The other reference-free losses.

Frequently asked questions#

References#

1. Meng, Y., Xia, M., Chen, D. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. arxiv.org/abs/2405.14734
2. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
3. Park, R., Rafailov, R., Ermon, S., Finn, C. (2024). Disentangling Length from Quality in Direct Preference Optimization. arxiv.org/abs/2403.19159
4. Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T. B. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arxiv.org/abs/2404.04475
5. Lambert, N. et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arxiv.org/abs/2411.15124
6. Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., Kim, Y. J. (2024). Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. arxiv.org/abs/2401.08417
7. Hong, J., Lee, N., Thorne, J. (2024). ORPO: Monolithic Preference Optimization without Reference Model. arxiv.org/abs/2403.07691
8. Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., Liu, P. J. (2023). SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arxiv.org/abs/2305.10425
9. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (CPOTrainer, loss_type='simpo'). github.com/huggingface/trl

Related reading