SPPO and Nash learning from human feedback, explained

SPPO (Self-Play Preference Optimization) and Nash learning from human feedback (NLHF) treat alignment as a game rather than a reward-maximisation problem. Instead of fitting a scalar reward model and optimising it, they work with a preference model $\mathbb{P}(y\succ y^{\prime }\mid x)$, the probability that one response beats another, and look for the policy that no other policy beats more than half the time: the Nash equilibrium of a two-player constant-sum game, also called the von Neumann winner. NLHF (Munos et al., 2023) established the framework and an algorithm, Nash-MD, with last-iterate convergence. SPPO (Wu et al., 2024) turned the idea into a simple iterative recipe: sample responses from the current model, estimate each one's win rate against the model itself, and regress the log-probability ratio onto that win rate. Both sit in the preference-tuning stage after SFT, in the slot occupied by iterative DPO, online DPO and PPO-RLHF.

Why a game: the problem with reward models#

Standard RLHF assumes preferences come from a Bradley-Terry model: each response has a hidden score $r(x,y)$ and $\mathbb{P}(y\succ y^{\prime })=\sigma (r(y)-r(y^{\prime }))$. That assumption forces preferences to be transitive. If A beats B and B beats C, A must beat C. Real preference data need not be. Individual raters can be inconsistent, and even when every individual is transitive, the population average can cycle; the NLHF paper's appendix constructs exactly such an aggregate from individually transitive preferences. A reward model fitted to cyclic data has to put the responses on a line anyway, and the policy that maximises it can be beaten.

Nash learning from human feedback was introduced by Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland and colleagues at Google DeepMind in December 2023. Its proposal: learn a pairwise preference model directly (a model that reads a prompt and two responses and outputs a probability), and train the policy towards the Nash equilibrium of that preference. The paper proves existence and uniqueness of a KL-regularised equilibrium, gives the Nash-MD algorithm with $O(1/T)$ last-iterate convergence in the tabular case, and demonstrates policy-gradient versions on a summarisation task, evaluated with PaLM 2 as the judge.

Self-Play Preference Optimization came from Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang and Quanquan Gu at UCLA and CMU in May 2024. It drops the policy-gradient machinery in favour of an offline regression per iteration, uses the 0.4B PairRM preference model, and needs only prompts. Starting from Mistral-7B-Instruct-v0.2 and three iterations on 60K UltraFeedback prompts (20K per iteration), it raised the AlpacaEval 2.0 length-controlled win rate from 17.11% to 28.53%, against 22.30% for iterative DPO and 20.06% for iterative IPO under the same protocol; from Llama-3-8B-Instruct it went from 22.92% to 38.77%.

The game and its solution#

Define the win rate of a response against a policy and of one policy against another:

Because $\mathbb{P}(y\succ y^{\prime })+\mathbb{P}(y^{\prime }\succ y)=1$, the game in which player one picks $\pi$, player two picks ${\pi }^{\prime }$, and player one's payoff is $\mathbb{P}(\pi \succ {\pi }^{\prime })$ is constant-sum and symmetric. Its Nash equilibrium is

and it guarantees $\mathbb{P}({\pi }^{\ast }\succ {\pi }^{\prime })\ge \frac{1}{2}$ against every ${\pi }^{\prime }$. Under Bradley-Terry preferences the equilibrium puts its mass on the highest-reward response, so nothing is lost when preferences are transitive. When they are not, the equilibrium is a mixture. NLHF adds KL regularisation towards a reference $\mu$ (the SFT model) with strength $\tau$:

which has a unique equilibrium ${\pi }_{\tau }^{\ast }$. It is the game-theoretic analogue of RLHF's KL-regularised reward maximisation.

Nash-MD: mirror descent against a mixture

Nash-MD updates the policy against a geometric mixture of itself and the reference:

With ${\eta }_t=2/(\tau (t+2))$, $\mathrm{KL}({\pi }_{\tau }^{\ast },{\pi }_T)\le 8/({\tau }^2(T+1))$: the last iterate converges, so you never need to store or sample from a mixture of past policies. For neural policies, the paper's Nash-MD-PG samples one response from ${\pi }_{\theta }$ and an opponent response from a geometric mixture of ${\pi }_{\theta }$ and $\mu$ (mixture weight $\beta$), queries the preference model, and uses the result as a reward in a regularised policy gradient. On TL;DR summarisation, mixture weights between 0.125 and 0.375 beat both pure self-play ($\beta =0$) and best response against the SFT model ($\beta =1$).

SPPO: the objective#

SPPO starts from the classic multiplicative-weights update for solving such games (Freund and Schapire, 1999):

Responses that beat the current policy on average gain probability; responses that lose to it lose probability. With $\eta =\Theta (1/\sqrt{T})$, the average of the iterates ${\bar{\pi }}_T=\frac{1}{T}{\sum }_t{\pi }_t$ has duality gap $O(1/\sqrt{T})$. DPO and IPO would cancel $\log Z$ by taking differences between two responses. SPPO instead fits the equation for each response directly, with a squared loss, and replaces $\log Z$ by the constant $\eta /2$ (its limit when pairwise outcomes are 0 or 1 with equal chance):

The win rate is estimated from $K$ samples per prompt, comparing each with all $K$ (itself included, which contributes $\frac{1}{2}$):

A tie, $\hat{P}=\frac{1}{2}$, means no change. The reference in the log-ratio is the previous iterate ${\pi }_t$, not the SFT model; every iteration re-anchors.

Why it behaves like policy gradient

The gradient of the squared loss at the start of an iteration, where ${\pi }_{\theta }={\pi }_t$ and the log-ratio is 0, is

which is a REINFORCE step with reward $\hat{P}$ and baseline $\frac{1}{2}$. The paper makes this connection explicit: the win rate is the reward, ${\eta }^{-1}\log Z_{{\pi }_t}(x)$ is the ideal baseline (the soft value), and $\frac{1}{2}$ is a cheap approximation to it. As the log-ratio grows, the square loss also stops the update when the target is reached, which is what keeps SPPO near its anchor.

Next to DPO, IPO and KTO

With hard labels ($\mathbb{P}(y_w\succ y_l)=1$) and only the winner and loser, write $\beta =1/\eta$, $a=\beta \log \frac{{\pi }_{\theta }(y_w)}{{\pi }_{\text{ref}}(y_w)}$, $b=\beta \log \frac{{\pi }_{\theta }(y_l)}{{\pi }_{\text{ref}}(y_l)}$. Then

DPO and IPO constrain only the gap $a-b$; both log-ratios can fall together, which is the likelihood displacement seen with DPO. SPPO pins each one separately: the winner's log-ratio is pushed towards $+\frac{1}{2}$ (in $\beta$ units) and the loser's towards $-\frac{1}{2}$, so the winner's likelihood must rise. This is the "hard" SPPO loss that TRL implements as loss_type="sppo_hard".

Worked example: one prompt, three samples#

Sample $K=3$ responses from ${\pi }_t$. The preference model gives $\mathbb{P}(y_1\succ y_2)=0.8$, $\mathbb{P}(y_1\succ y_3)=0.6$, $\mathbb{P}(y_2\succ y_3)=0.3$. Including each response's tie with itself:

With the paper's $\eta =1000$, the regression targets for the log-ratios are $1000\times (0.633-0.5)=133.3$, $-166.7$ and $33.3$ nats. Following the paper's selection, the winner is $y_1$ and the loser $y_2$. At the start of the iteration both log-ratios are 0, so the loss is $133.3^2+166.7^2=45,556$, and the gradient coefficients on $\nabla \log {\pi }_{\theta }$ are $-2(0-133.3)=266.7$ towards raising $y_1$ and $-333.3$, lowering $y_2$. These targets are large on purpose. A sequence log-ratio of 133 nats is about 0.27 nats per token on a 500-token response, and one epoch at learning rate $5\times 10^{-7}$ gets nowhere near it, so in practice the loss works in its policy-gradient regime: the size of each response's push is proportional to how far its win rate is from one half. Response $y_3$, barely above a coin flip, would get a push one fifth as large as $y_1$'s if it were included.

Compare TRL's hard-label version with $\beta =0.001$: the targets become $\pm 0.5/\beta =\pm 500$ nats regardless of how decisive the preference was. The soft version carries more information per pair; the paper found the result fairly robust to how the win rate is estimated (using only $K=2$ samples gave 28.26% length-controlled win rate at iteration 3 versus 28.53% with $K=5$, with less length growth).

Toy run: cycles, averages and the last iterate#

We computed everything exactly on the four-response game in Figure 1 (preference matrix: A beats B 0.9, B beats C 0.6, C beats A 0.7; D beats B 0.6, loses to A 0.6 and C 0.65). A linear program gives the Nash equilibrium: A 0.143, B 0.286, C 0.571, D 0, with game value exactly $\frac{1}{2}$. Fitting Bradley-Terry scores to the same matrix gives A 0.31, C 0.26, D $-0.15$, B $-0.41$. We measure each policy by its exploitability, the win rate of the best response against it minus $\frac{1}{2}$.

Two points come out of this. First, the reward-model route gets this game wrong in a specific way: the KL-regularised RLHF optimum with $\beta =0.1$ can be beaten 62.6% of the time, because the fitted reward had to rank a cycle. Even the uniform policy is harder to exploit. Second, the theory behind SPPO is about the average. The exact multiplicative-weights update's average reached exploitability 0.0034 after 1,000 steps, but its last iterate cycled ever wider and sat at 0.389, almost all of its mass on B. At $\eta =0.3$ the last iterate still reached 0.233 after step 500. Nash-MD, which plays against a geometric mixture with the reference, has a last iterate that converges: with $\tau =0.1$ to a regularised equilibrium at 0.053, with $\tau =0.02$ to 0.0088, close to the true Nash mixture. SPPO in practice uses the last iterate, three iterations deep, each anchored to the previous model by the squared loss. On real preference data that is mostly transitive, the cycling problem is mild; this toy shows why theory recommends anchoring, regularisation or averaging when it is not.

The SPPO algorithm#

policy = load("mistral-7b-instruct")                     # pi_1
for t in range(3):                                       # 3 iterations, new prompt split each
    prompts = ultrafeedback_split[t]                     # 20K prompts
    data = []
    for x in prompts:
        ys = policy.sample(x, n=5, temperature=1.0, top_p=1.0)
        Pm = [[pair_rm(x, a, b) for b in ys] for a in ys]          # K x K, O(K^2) calls
        win = [sum(row) / len(ys) for row in Pm]                   # P_hat(y_i > pi_t)
        w, l = argmax(pairrm_score(ys)), argmin(pairrm_score(ys))  # keep winner and loser
        data += [(x, ys[w], win[w]), (x, ys[l], win[l])]
    ref = freeze(copy(policy))                           # pi_t is the anchor
    for batch in loader(data, batch_size=64):            # lr 5e-7, linear schedule, warm-up 0.1
        logratio = policy.seq_logprob(batch.x, batch.y) - ref.seq_logprob(batch.x, batch.y)
        loss = ((logratio - eta * (batch.win - 0.5)) ** 2).mean()  # eta = 1e3
        loss.backward(); opt.step(); opt.zero_grad()

Each iteration has three phases: generate $K$ responses per prompt, score all $K^2$ pairs with the preference model, and train. The paper keeps only the highest- and lowest-scoring responses to match iterative DPO's data budget, which is a choice, not a requirement; the loss is defined for every sampled response. Data is prompts only, plus a preference model; the paper used UltraFeedback's roughly 60K prompts split into three disjoint 20K portions, one per iteration, following the Snorkel iterative-DPO setup it compared against.

The preference model

SPPO's advantage over reward-model methods only exists if the judge can express non-transitive preferences. PairRM, a 0.4B DeBERTa-v3-based model, reads both responses and outputs a relative score $s(y,y^{\prime };x)$, with $\mathbb{P}(y\succ y^{\prime })=\sigma (s)$; its scores are not constrained to be additive, so $s(y_1,y_2)+s(y_2,y_3)\ne s(y_1,y_3)$ in general. An LLM judge prompted with both responses has the same property. A Bradley-Terry reward model does not, and using one with SPPO reduces it to a squared-loss variant of reward optimisation.

Hyperparameters#

Selection matters as much as tuning. The paper selected hyperparameters by average PairRM win rate on a held-out UltraFeedback subset, which risks optimising the judge rather than quality. Keep a second, independent evaluation (a different judge, or humans) and a length-controlled metric.

Compute and memory#

Per iteration, SPPO's training phase costs what DPO costs: a trainable policy (about 16 bytes per parameter with Adam in mixed precision) plus a frozen anchor ${\pi }_t$ (2 bytes per parameter in bf16, or precomputed log-probabilities), with a forward and backward pass per response. Generation is $K$ responses per prompt, and judging is $K^2$ preference-model calls per prompt; with $K=5$ and a 0.4B PairRM that is 25 cheap calls, but with a large LLM judge the $K^2$ term can dominate:

The paper ran on 8 A100 GPUs. Nash-MD-PG, being online policy gradient, costs what PPO or GRPO cost per step, plus sampling the opponent from the mixture policy, which needs both ${\pi }_{\theta }$ and $\mu$ available at generation time.

Failure modes#

Length growth. SPPO's responses lengthened across iterations (1,676 to 2,163 characters on AlpacaEval for Mistral), less than iterative DPO's or IPO's in the same study but still present. Judges, PairRM included, tend to favour longer answers. Track length per iteration and report length-controlled win rates.

Judge exploitation. Self-play against a fixed judge is still optimisation against a fixed model. If the PairRM win rate keeps climbing while an independent evaluation stalls, you are fitting the judge. The paper also reports best-of-16 reranking with PairRM on top of SPPO models, which compounds the same judge.

Last-iterate cycling. Theory guarantees the average policy, and our toy shows the last iterate of the idealised update can cycle when preferences are intransitive. Watch the win rate of each iterate against the previous two, not only the last: if ${\pi }_3$ beats ${\pi }_2$, ${\pi }_2$ beats ${\pi }_1$, but ${\pi }_1$ beats ${\pi }_3$, you are in a cycle and should regularise towards the SFT model (as Nash-MD does) or average checkpoints.

Win-rate saturation. As the policy improves, all samples look alike to the judge, $\hat{P}\to \frac{1}{2}$ for everyone, and the gradient vanishes. Diverse sampling and harder prompts help; so does a stronger judge.

Over-anchoring. Re-anchoring to ${\pi }_t$ each iteration means drift from the SFT model is never directly penalised. Over many iterations, measure KL to the original SFT model and capability regressions on held-out benchmarks.

When to use SPPO or NLHF#

Minimal implementation#

import torch

def win_rates(pref_matrix):
    """pref_matrix: (K, K) with P[i, j] = P(y_i beats y_j), diagonal 0.5. Returns (K,)."""
    return pref_matrix.mean(dim=1)

def sppo_loss(logp, logp_anchor, win, eta=1e3):
    """logp, logp_anchor: (N,) summed sequence log-probs under pi_theta and pi_t; win: (N,) P_hat."""
    logratio = logp - logp_anchor
    target = eta * (win - 0.5)
    return ((logratio - target) ** 2).mean()

def sppo_hard_loss(logp_w, ref_w, logp_l, ref_l, beta=1e-3):
    """Hard-label pairwise form, as in TRL's 'sppo_hard'."""
    a = logp_w - ref_w; b = logp_l - ref_l
    return ((a - 0.5 / beta) ** 2 + (b + 0.5 / beta) ** 2).mean()

def nash_md_mixture_logits(logits_policy, logits_ref, beta):
    """Geometric mixture of next-token distributions: one way to sample a Nash-MD-PG opponent."""
    return torch.log_softmax((1 - beta) * logits_policy.log_softmax(-1) + beta * logits_ref.log_softmax(-1), -1)

In TRL, DPOConfig(loss_type="sppo_hard", beta=0.001) gives the hard-label SPPO loss inside DPOTrainer; the soft, win-rate version needs the per-response probabilities computed outside the trainer, as the paper's code does. TRL also shipped a NashMDTrainer up to v1.13; it is no longer in the library but remains in its git history. In POFE, SPPO runs as an iterative preference recipe: each round's generation, judging and training are separate stages, and the report includes a cross-iteration win-rate table so cycles are visible.

Further reading#

Munos et al. (2023) for the NLHF framework, the regularised game and Nash-MD. Wu et al. (2024) for SPPO, its policy-gradient interpretation and the comparison with DPO, IPO and KTO. Swamy et al. (2024) for the minimaximalist view, which reaches a similar self-play objective from a different direction, and Rosset et al. (2024) for Direct Nash Optimization, a regression-based alternative with an iterative contrastive objective. Azar et al. (2023) for IPO and the general $\Psi$PO framework, and Jiang et al. (2023) for LLM-Blender, where PairRM comes from.

Frequently asked questions#

References#

1. Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, Quanquan Gu (2024). Self-Play Preference Optimization for Language Model Alignment. arxiv.org/abs/2405.00675
2. Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, et al. (2023). Nash Learning from Human Feedback. arxiv.org/abs/2312.00886
3. Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, Alekh Agarwal (2024). A Minimaximalist Approach to Reinforcement Learning from Human Feedback. arxiv.org/abs/2401.04056
4. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, Tengyang Xie (2024). Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences. arxiv.org/abs/2404.03715
5. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arxiv.org/abs/2310.12036
6. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
7. Dongfu Jiang, Xiang Ren, Bill Yuchen Lin (2023). LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. arxiv.org/abs/2306.02561
8. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (DPOTrainer, sppo_hard loss). github.com/huggingface/trl

Related reading