IPO (Identity Preference Optimization), explained

Identity Preference Optimization (IPO) is a preference-tuning loss that looks like DPO with one change: instead of pushing the gap between chosen and rejected log-likelihood ratios through a logistic loss, it regresses that gap onto a fixed target with a squared error. The target is $1/(2\tau )$, where $\tau$ is the regularization strength. That small edit matters when preferences in the data are deterministic or nearly so, which is most real datasets: DPO keeps widening the gap indefinitely and the KL penalty stops binding, while IPO stops at the target. IPO occupies the same slot in the stack as DPO, after SFT, trained offline on (prompt, chosen, rejected) triples with a frozen reference model.

Where IPO comes from#

IPO was introduced by Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko and Rémi Munos (Google DeepMind) in A General Theoretical Paradigm to Understand Learning from Human Preferences, posted in October 2023 and published at AISTATS 2024. The paper is mostly theory. It asks what DPO and RLHF actually optimize once you stop assuming a perfect Bradley-Terry reward, finds a failure mode, and proposes IPO as the simplest objective that avoids it.

The authors point at two approximations in standard RLHF. First, pairwise preferences are replaced by pointwise rewards through the Bradley-Terry model. Second, a reward model trained on that substitution is assumed to generalize to the policy's own outputs. DPO removes the second approximation but keeps the first, and the first is where the trouble lives.

The ΨPO objective#

Let $p^{\ast }(y\succ y^{\prime }\mid x)$ be the true probability that response $y$ is preferred to $y^{\prime }$ for prompt $x$, $\mu$ the distribution that generated the comparison responses, ${\pi }_{\text{ref}}$ the reference policy and $\tau >0$ a regularization weight. For any non-decreasing function $\Psi :[0,1]\to \mathbb{R}$, Azar et al. define

In words: maximize a transformed win rate against the data distribution, while staying close to the reference. If $\Psi (q)=\log \frac{q}{1-q}$ and preferences follow Bradley-Terry, $\Psi (p^{\ast }(y\succ y^{\prime }))=r(y)-r(y^{\prime })$, and this objective has the same optimum as RLHF and DPO (their Proposition 1, with $\tau$ playing the role of DPO's $\beta$). The optimal policy has the familiar form

Why the logit Ψ overfits

Suppose $y$ always beats $y^{\prime }$: $p^{\ast }(y\succ y^{\prime })=1$. Bradley-Terry then needs $r(y)-r(y^{\prime })\to +\infty$, and plugging that into the optimal policy gives ${\pi }^{\ast }(y^{\prime })=0$ for any value of $\tau$. The KL term still appears in the objective, but it can no longer hold the policy anywhere near the reference. The paper's summary: the strength of the KL regularization becomes weaker and weaker the more deterministic the preferences.

Finite data makes it worse. Even if the true preference is 0.8, a pair that appears once or twice in the dataset has an empirical preference of exactly 1. Most prompts in a real preference dataset are compared exactly once, so almost every empirical preference is 0 or 1. Standard RLHF survives this in practice because its reward model is trained with regularization and early stopping and therefore underfits: it never assigns the infinite reward gaps the data technically asks for. DPO has no reward model to underfit, so it loses that implicit protection.

The IPO loss#

Setting $\Psi$ to the identity bounds the quantity being maximized. Azar et al. derive a sampled loss for this case that needs only preference pairs. Define the log-likelihood-ratio gap

The IPO objective, starting from $\pi ={\pi }_{\text{ref}}$, is

Compare DPO, written with the same $h$:

Both see each pair only through $h$. DPO's loss decreases monotonically in $h$, so it always prefers a larger gap. IPO's loss is minimized at $h=1/(2\tau )$ and rises on either side, so a gap that overshoots the target is pulled back. The gradients make the difference concrete:

The IPO derivation also uses a symmetry trick worth knowing: each observed pair contributes both $(y_w,y_l)$ with label 1 and $(y_l,y_w)$ with label 0, which halves the variance of the estimate and collapses, up to a constant, to the single squared term above. The paper's interpretation is that IPO regresses the gap between the policy's and the reference's log-likelihood ratios onto ${\tau }^{-1}/2$; the weaker the regularization (smaller $\tau$), the larger the gap it asks for.

Where IPO sits among its siblings

Tang et al. (2024) unify DPO, IPO and SLiC as one family: apply a convex function $f$ to $\beta h$. DPO uses the logistic loss $f(z)=\log (1+e^{-z})$, SLiC the hinge $\max (0,1-z)$, IPO the squared loss $(z-1{)}^2$ up to rescaling. Their analysis shows how the shape of $f$ determines how offline losses regularize, and that this offline regularization is related to, but not the same as, the KL penalty in the RLHF objective. Calandriello et al. (2024) go further and show that an online version of IPO (pairs sampled from the current policy) is equivalent to Nash Mirror Descent, a self-play algorithm for general, non-Bradley-Terry preferences. That makes IPO a natural loss for online and iterative preference training.

Worked example by hand#

Take $\tau =0.1$, so the target margin is $1/(2\times 0.1)=5$ nats. Use the same pair as in our DPO article: policy log-probs $-42.0$ (chosen) and $-50.0$ (rejected), reference $-43.5$ and $-48.0$.

• Log-ratio of chosen: $-42.0-(-43.5)=1.5$. Log-ratio of rejected: $-50.0-(-48.0)=-2.0$. Gap $h=3.5$.
• IPO loss $(3.5-5{)}^2=2.25$. Gradient $\partial \mathcal{L}/\partial h=2(3.5-5)=-3$: increase the gap.
• Suppose training continues and the gap reaches $h=8$. IPO loss $(8-5{)}^2=9$, gradient $+6$: decrease the gap.
• DPO with $\beta =0.1$ at $h=8$: loss $\log (1+e^{-0.8})=0.371$, gradient $-0.1\sigma (-0.8)=-0.031$: still pushing the gap wider, gently but forever.

Now a case where the closed form is known. Three actions $a,b,c$, uniform reference, and a dataset with each pair observed once in a strict order: $a\succ b$, $b\succ c$, $a\succ c$. With uniform reference, $h$ is just the difference of log-probabilities. Let $u=\log \pi (a)-\log \pi (b)$ and $v=\log \pi (b)-\log \pi (c)$, with target $t=1/(2\tau )$. IPO minimizes $(u-t{)}^2+(v-t{)}^2+(u+v-t{)}^2$; by symmetry $u=v$, and setting the derivative to zero gives $u=2t/3$. For $\tau =1$ ($t=0.5$), $u=1/3$, so $\pi \propto (1,e^{-1/3},e^{-2/3})=(0.448,0.321,0.230)$. For $\tau =0.1$ ($t=5$), $u=10/3$, giving $\pi (a)=0.965$. DPO on the same data drives $\pi (a)\to 1$ for every $\beta$.

Toy runs#

Replicating the three-action bandit

We reproduced the setup from Section 5.3 of the IPO paper: softmax policy over three actions, uniform reference, dataset $\{(a,b),(b,c),(a,c)\}$, Adam with learning rate 0.01, minibatches of 9 sampled with replacement, 18,000 steps. We ran DPO with $\beta$ and IPO with $\tau$ at the same four values, three seeds each.

DPO ignores its regularization entirely and becomes greedy, which is exactly what the paper reports. IPO lands on the closed-form optimum (computed from $u=2t/3$) to within about 0.005, and $\tau$ now does what a regularizer should: larger $\tau$ keeps the policy closer to uniform.

Deterministic labels with shared parameters

The bandit has no parameter sharing. To check that the effect survives generalization, we used a log-linear policy over 200 responses with 16 shared features (the same world as in our DPO article), a deterministic annotator (the higher true reward always wins) and only 400 pairs sampled from the reference (358 after removing pairs that drew the same response twice). Adam, learning rate 0.01, batch 32, 8,000 steps.

With deterministic labels, raising DPO's $\beta$ from 0.1 to 0.5 did not reduce drift; KL kept climbing in both runs and ended slightly higher at $\beta =0.5$. The mean margin of 28.5 nats is far beyond anything the data supports. IPO's KL plateaus within the first 2,000 steps and stays there, at a level set by $\tau$. One caveat for reading this honestly: in this toy the true reward is exactly the direction the preferences reveal, so drift happens to help; the DPO runs reached a higher expected true reward (2.99 and 3.01, against 2.05 for IPO at τ = 0.1 and a reference value of 0.21). The point is control. In a real model, drift you did not ask for is where formatting breaks and capabilities regress, and IPO gives you a dial that actually limits it.

The algorithm and data#

Input: SFT model, pairs D = {(x, y_w, y_l)}, tau
pi_ref = frozen copy; pi_theta = trainable copy
target = 1 / (2 * tau)
for each minibatch:
    h = (logp(pi_theta, y_w) - logp(pi_ref, y_w)) - (logp(pi_theta, y_l) - logp(pi_ref, y_l))
    loss = mean((h - target) ** 2)
    step optimizer
    log: mean h, fraction of pairs with h > target, KL estimate on samples, eval metric

The data is identical to DPO's: one prompt, two responses, a binary label. Anything that works for DPO (human comparisons, LLM-judge rankings, reward-model-scored samples from your own model) works for IPO, and you can switch between the two losses on the same dataset with one flag. IPO is most attractive exactly where DPO is weakest: small datasets, a single comparison per prompt, and judges that produce confident, near-deterministic labels.

The length question

The paper's $h$ uses full-sequence log-probabilities, so the margin and therefore the right $\tau$ scale with response length. Hugging Face TRL's implementation divides each log-ratio by the number of completion tokens before computing the squared loss; the code notes that this length normalization is not explicitly discussed in the paper and that the TRL maintainers confirmed the choice with the IPO authors. In practice this means two IPO runs with the "same" $\tau$ can behave very differently depending on the implementation. Check which one you are using before copying a value.

Hyperparameters#

τ (the target). Choose $\tau$ by thinking about the margin you want, not by analogy with DPO's $\beta$. With per-token averaged log-ratios, a target of $1/(2\tau )=0.5$ nats per token ($\tau =1$) is already a large separation; with summed log-ratios over a few hundred tokens, targets of 5 to 50 nats ($\tau$ between 0.1 and 0.01) are plausible. For reference points: the OAIF paper used $\tau =1.0$ for IPO, and the SimPO authors searched $\tau \in \{0.01,0.1,0.5,1.0\}$ for their IPO baselines. Sweep at least three values spaced by about 3x and pick on a downstream metric.

Learning rate and epochs. Start where you would for DPO (around 5e-7 to 1e-6 for full fine-tuning at 7B scale) and adjust. The squared loss produces gradients proportional to the distance from target, which can be large early on when the target is big, so keep warmup. Because IPO has a stable optimum, it is less sensitive to training for extra epochs than DPO; watching the fraction of pairs above target tells you when it has converged.

Batch size. The usual 64 to 128 pairs. The loss is a regression, so noisy small batches show up as jitter in the mean margin rather than as instability.

Compute and memory#

Exactly DPO's. Per pair: forward and backward through the policy on two sequences, forward through the frozen reference on the same two (or a single precomputation pass). Memory is roughly 16 bytes per parameter for the trainable policy with Adam in mixed precision, plus 2 bytes per parameter for a bf16 reference, plus activations for two sequences per example. With LoRA the reference costs nothing extra because it is the base model with adapters switched off. Switching from DPO to IPO is free in compute terms, which makes it a cheap ablation.

Failure modes#

Target too large. If $1/(2\tau )$ is bigger than any gap the data can support, IPO behaves like a DPO with an odd loss shape: margins climb for the whole run and KL drifts. Diagnose by plotting the distribution of $h$ against the target. If almost no pairs reach it after an epoch, $\tau$ is too small.

Target too small. The policy barely moves, training accuracy rises to the high 90s quickly (any positive margin counts as correct) and downstream metrics do not change. In our toy, $\tau =0.5$ held KL at 0.13 nats and expected true reward rose only from 0.21 (the reference) to 0.60, against 2.05 at $\tau =0.1$.

Pushing down good margins. Pairs whose gap already exceeds the target receive gradient that shrinks it, including pairs that are clean and easy. That is the intended regularization, but on a dataset mixing trivially easy and hard pairs, a single target can be too tight for some and too loose for others.

Label noise. Squared loss treats a mislabeled pair as an instruction to reach $+1/(2\tau )$ in the wrong direction. The loss is bounded per pair only by the reachable margin, so a few percent of flipped labels contribute disproportionately at large $h$. Filter obvious ties and disagreements before training.

Implementation mismatch. Summed versus length-averaged log-ratios changes the meaning of $\tau$ by a factor equal to the response length. It is an easy mistake to make when copying a $\tau$ from a paper or another codebase.

What a healthy run looks like

Because the policy starts equal to the reference, every gap starts at $h=0$ and the initial loss is exactly $(1/(2\tau ){)}^2$: 25 for $\tau =0.1$, 1 for $\tau =0.5$. That gives you a free sanity check on the first logged step; any other value means the reference or the masking is wrong. From there the mean margin should rise quickly and then flatten below the target, because pairs the model cannot separate (ties, label noise, near-duplicates) hold the average down while easy pairs sit at or slightly above it. The loss should flatten at a level set by the spread of margins around the target, not keep falling toward zero. A DPO run on the same data has no such equilibrium: its loss keeps creeping down as margins on easy pairs grow without bound. If an IPO run's mean margin crosses the target and keeps climbing, something other than the IPO term is driving it, usually an added NLL term with too large a weight or a learning rate high enough that the optimizer overshoots.

Metrics to log: mean and distribution of $h$ relative to target, chosen and rejected log-ratios separately, KL to the reference on generated samples, output length, and a task metric.

When to use IPO#

Published comparisons do not show IPO winning consistently on chat benchmarks. In the SimPO paper's tuned baselines, IPO was competitive with DPO in some settings and behind in others; OAIF found the online versions of DPO, IPO and SLiC all improved over their offline counterparts. The honest framing is that IPO is the better-behaved loss when your labels are close to deterministic, and an easy ablation otherwise. In POFE it is one option of the preference recipe, sharing data and evaluation with the DPO run so the comparison is apples to apples.

Minimal implementation#

import torch

def ipo_loss(logp_w, logp_l, ref_logp_w, ref_logp_l, tau=0.1,
             len_w=None, len_l=None):
    """logp_*: summed response log-probs, shape [B].
    Pass token counts len_w, len_l to use the length-averaged convention."""
    rw = logp_w - ref_logp_w
    rl = logp_l - ref_logp_l
    if len_w is not None:
        rw, rl = rw / len_w, rl / len_l
    h = rw - rl
    loss = ((h - 1.0 / (2.0 * tau)) ** 2).mean()
    return loss, {"margin": h.mean().item(),
                  "frac_above_target": (h > 1.0 / (2.0 * tau)).float().mean().item()}

In TRL, IPO is a loss type on the DPO trainer, with beta playing the role of $\tau$ and the length-averaged convention described above:

from trl import DPOConfig, DPOTrainer

args = DPOConfig(output_dir="ipo-out", loss_type="ipo", beta=0.5,
                 learning_rate=5e-7, num_train_epochs=1, bf16=True)
trainer = DPOTrainer(model=model, args=args, train_dataset=pairs,
                     processing_class=tokenizer)
trainer.train()

Further reading#

Start with the IPO paper's Sections 4 and 5 for the overfitting argument and the sampled loss; the others place IPO in the wider family.

• Gheshlaghi Azar et al. (2023), A General Theoretical Paradigm to Understand Learning from Human Preferences. ΨPO, the overfitting argument and IPO.
• Rafailov et al. (2023), Direct Preference Optimization. The method IPO modifies.
• Tang et al. (2024), Generalized Preference Optimization. DPO, IPO and SLiC as convex losses on the same margin.
• Calandriello et al. (2024), Human Alignment of Large Language Models through Online Preference Optimisation. Online IPO and its link to Nash Mirror Descent.
• Guo et al. (2024), Direct Language Model Alignment from Online AI Feedback. Online DPO, IPO and SLiC with an LLM annotator.
• Meng et al. (2024), SimPO. Includes tuned IPO baselines and search ranges.

Frequently asked questions#

References#

1. Gheshlaghi Azar, M., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., Munos, R. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arxiv.org/abs/2310.12036
2. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
3. Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Ávila Pires, B., Piot, B. (2024). Generalized Preference Optimization: A Unified Approach to Offline Alignment. arxiv.org/abs/2402.05749
4. Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Avila Pires, B., Richemond, P. H., Le Lan, C., Valko, M., Liu, T., Joshi, R., Zheng, Z., Piot, B. (2024). Human Alignment of Large Language Models through Online Preference Optimisation. arxiv.org/abs/2403.08635
5. Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., Blondel, M. (2024). Direct Language Model Alignment from Online AI Feedback. arxiv.org/abs/2402.04792
6. Meng, Y., Xia, M., Chen, D. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. arxiv.org/abs/2405.14734
7. Bradley, R. A., Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. doi.org/10.2307/2334029
8. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (DPOTrainer, loss_type='ipo'). github.com/huggingface/trl

Related reading