SPIN explained: self-play fine-tuning for language models

SPIN (Self-Play fIne-tuNing) improves a supervised fine-tuned model using nothing but the SFT data it was already trained on. In each iteration the current model generates its own answers to the SFT prompts, and the next model is trained, with a DPO-style pairwise loss, to prefer the human-written answer over its own. The previous iteration's model serves both as the source of "rejected" responses and as the reference policy. No preference labels, no reward model and no stronger teacher are needed. SPIN sits between SFT and preference optimisation in the post-training stack: it is a way to keep extracting signal from a demonstration dataset after plain SFT has stopped improving on it.

Origin and the problem it solves#

SPIN was introduced by Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji and Quanquan Gu (UCLA) in Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, posted in January 2024 and published at ICML 2024. Their starting point was a practical observation. Take zephyr-7b-sft-full, a Mistral-7B model already fine-tuned on the UltraChat 200K dialogue set (Tunstall et al., 2023), and keep training it with SFT on the same data: benchmark scores plateau or fall. The data still contains information the model has not absorbed, but maximum likelihood is no longer a good way to get at it.

The idea is borrowed from adversarial training (Goodfellow et al., 2014) and self-play. If the model's own outputs are distinguishable from human outputs, a discriminator can say how, and that difference is a training signal. SPIN's twist is that the discriminator and the generator are the same language model at consecutive iterations, and the discriminator's score is the log-probability ratio between them, so the "discriminator" can be trained directly as a new policy.

The paper reports that on the Open LLM Leaderboard average, SPIN lifted zephyr-7b-sft-full from 58.14 to 60.80 after iteration 0, 62.12 after iteration 1, 62.97 after iteration 2 and 63.16 after iteration 3, using a 50K-prompt subset of UltraChat 200K. MT-Bench rose from 5.94 to 6.78. For comparison, zephyr-7b-dpo-full, trained with an additional 62K GPT-4-labelled preference pairs, averaged 61.31.

The objective#

Let $x$ be a prompt from the SFT set and $y\sim p_{\text{data}}(\cdot \mid x)$ its human response. Let $p_{{\theta }_t}$ be the model after iteration $t$ and $y^{\prime }\sim p_{{\theta }_t}(\cdot \mid x)$ a response it generates. SPIN frames iteration $t+1$ as a two-player game. The main player is a function $f(x,y)$ that should score human responses above the opponent's:

where $\ell$ is a convex, decreasing loss; the paper uses the logistic loss $\ell (t)=\log (1+e^{-t})$. The opponent then updates to the policy that $f_{t+1}$ rewards most, subject to staying close to $p_{{\theta }_t}$ in KL with strength $\lambda$. Solving that inner problem in closed form gives $f_{t+1}(x,y)=\lambda \log \frac{p_{{\theta }_{t+1}}(y\mid x)}{p_{{\theta }_t}(y\mid x)}$. Substituting it back collapses both players into one loss over the new parameters $\theta$:

Write the margin as $m=\lambda [(\log p_{\theta }(y)-\log p_{{\theta }_t}(y))-(\log p_{\theta }(y^{\prime })-\log p_{{\theta }_t}(y^{\prime }))]$. With the logistic loss the gradient is

which raises the likelihood of the human response, lowers that of the self-generated one, and stops pushing on a pair once its margin is large ($\sigma (-m)\to 0$).

SPIN is DPO with a moving reference and synthetic negatives

Put the two losses side by side:

Since $\log (1+e^{-m})=-\log \sigma (m)$, they are the same function. The differences are entirely in the data and the schedule. In DPO, $y_w$ and $y_l$ are two model responses ranked by a person or judge, and ${\pi }_{\text{ref}}$ is fixed. In SPIN, $y_w$ is always the human SFT response, $y_l$ is always a fresh sample from the previous iteration, and the reference is replaced every iteration. That last point is what makes SPIN a sequence of games rather than one optimisation.

Why it converges to the data distribution

Two results from the paper explain the behaviour. First (their Theorem 5.2), under mild conditions on $\ell$, the current model ${\theta }_t$ is a global minimiser of $L_{\text{SPIN}}(\cdot ;{\theta }_t)$ exactly when $p_{{\theta }_t}=p_{\text{data}}$. If the model already matches the human distribution, $y$ and $y^{\prime }$ are exchangeable and the expected gradient is zero; if it does not, some $\lambda$ makes a step profitable. Self-play therefore stops where SFT would ideally stop, at the data distribution, and not beyond it. Second (Theorem 5.4), for the logistic loss, if the global minimiser is reachable, the next model is

Each iteration multiplies the model's probabilities by the human-to-model ratio raised to $1/\lambda$. Responses the model under-produces go up; responses it over-produces go down. The exponent decides the step size, and that has a sharp consequence worked through below.

Worked example: one pair, and what \(\lambda\) does#

One pair. At the start of an iteration, $\theta ={\theta }_t$, so for any pair the log-ratios are zero, $m=0$, the loss is $\log 2=0.693$, and the gradient weight $\sigma (-m)$ is 0.5. Suppose a human response has sequence log-probability $-20.0$ under $p_{{\theta }_t}$ and the model's own response $-35.0$. After some training, the new model scores them at $-18.0$ and $-38.0$. With $\lambda =0.1$, the margin is $0.1\times [(-18+20)-(-38+35)]=0.1\times (2+3)=0.5$, the loss falls to $\log (1+e^{-0.5})=0.474$, and the gradient weight to $\sigma (-0.5)=0.378$. The pair still pushes, a little less hard.

The step size of an iteration. Take one prompt with three possible responses. Humans write them with probabilities $p_{\text{data}}=(0.6,0.3,0.1)$; the current model produces $(0.3,0.3,0.4)$, a total-variation distance of 0.30. Apply the closed-form update from Theorem 5.4 for several values of $\lambda$:

At the global optimum, $\lambda =1$ lands on the data distribution in one step, larger $\lambda$ takes partial steps, and smaller $\lambda$ overshoots, collapsing onto whatever the model under-produced. The paper's $\lambda =0.1$ works in practice because LLM training never reaches that optimum: two epochs at a learning rate of $5\times 10^{-7}$ is a small, early-stopped move, and sequence log-ratios sum over hundreds of tokens so the effective scale differs from this per-response picture. It also explains why the authors raised $\lambda$ to 5.0 in the final iteration, when the model is close to convergence and a smaller step is wanted.

Toy run: iterate gently or overshoot#

To see the dynamics with real optimisation rather than the closed form, we simulated one prompt with 30 possible responses. The human distribution is a random softmax; the "SFT model" starts under-confident and partly wrong (total-variation distance 0.42 from the human distribution). A fixed dataset of 2,000 human responses plays the role of the SFT set; each SPIN iteration samples 2,000 responses from the current model, pairs them randomly with the human ones, and minimises the logistic SPIN loss with Adam (learning rate 0.05) for a fixed number of steps. We compare against continuing SFT on the same 2,000 human responses, and against training iteration 0 for the same total number of steps without regenerating. 8 seeds; all numbers are the script's output.

Three lessons carry over. First, the theory is right about the direction: small $\lambda$ with long training per iteration overshoots, and the distance got worse, not better, with more steps. Second, regenerating the opponent is what rescues small $\lambda$: with 5 steps per iteration, eight iterations brought $\lambda =0.1$ to 0.084 while the same 40 steps on iteration 0 alone stayed at 0.346. That matches the paper's finding that training longer within iteration 0 cannot match moving to iteration 1. Third, and less flattering: in a model small enough for SFT to fit the data exactly, SFT wins. SPIN's case rests on the regime the paper describes, where SFT has plateaued on a real network and the self-generated negatives give the gradient something SFT's likelihood cannot.

The algorithm, step by step#

model_t = load("sft-checkpoint")                            # iteration 0 opponent and reference
D = load_sft_pairs()                                         # [(x, y_human)], e.g. 50K examples
prev_synthetic = []
for t in range(T):                                           # T = 3 or 4 in the paper
    synthetic = [(x, y, model_t.generate(x)) for x, y in D]  # one y' per prompt, from the frozen model_t
    data = synthetic + prev_synthetic                        # paper: 50K at iteration 0, 100K after
    model_next = copy(model_t)
    for epoch in range(2):
        for batch in loader(data):
            m = lam * ((model_next.logp(batch.y) - model_t.logp(batch.y))
                       - (model_next.logp(batch.y_prime) - model_t.logp(batch.y_prime)))
            loss = softplus(-m).mean()                       # log(1 + exp(-m))
            loss.backward(); opt.step(); opt.zero_grad()
    prev_synthetic, model_t = synthetic, model_next

Data. Only the SFT set: prompts with high-quality human or curated responses. The paper used 50K prompts sampled from UltraChat 200K and, from iteration 1 onward, combined the new synthetic pairs with the previous iteration's, giving 100K pairs per iteration. One generation per prompt per iteration is enough. The quality ceiling is the quality of the human responses: SPIN converges to the data distribution, so it cannot exceed what the demonstrations show. Generation. The opponent samples must come from the frozen previous iteration, not the model being trained, and should use the same prompt template as SFT.

Design choices inside the method#

The loss $\ell$. The convergence result holds for any convex, decreasing $\ell$; the paper names correlation, hinge and logistic losses as valid choices. Only the logistic loss turns SPIN into DPO's functional form, which is why it is the one used in practice: every DPO implementation, trick and diagnostic carries over. A hinge loss would stop pushing on a pair as soon as its margin crossed a threshold, a harder form of the self-limiting behaviour the logistic weight $\sigma (-m)$ provides.

The main player as a probability metric. The game is motivated by integral probability metrics: the main player looks for the function that best separates samples of $p_{\text{data}}$ from samples of $p_{{\theta }_t}$, and the size of that separation measures how far apart the two distributions are. Restricting the function class to log-probability ratios of the same model family is what makes the discriminator a usable policy. It also means the model can only notice differences it can express as a change in its own likelihood. Stylistic differences (length, formatting, hedging phrases) are easy to express, so SPIN tends to fix those first.

Reusing old synthetic data. From iteration 1 the paper trains on the current iteration's pairs plus the previous one's. The older negatives are further from the human data, so they are easy pairs with large margins and small gradient weights, but they keep the model from drifting back toward behaviours it has already been taught to avoid. If memory or time is tight, dropping them is the first simplification to try; watch for regressions on the earlier iteration's pair accuracy if you do.

Combining with preference data. SPIN and DPO are complementary rather than competing. The paper reports that running DPO on the SPIN iteration 3 model, using the same 62K preference pairs Zephyr used, raised the leaderboard average from 63.16 to 64.05. A sensible pipeline is SFT, then SPIN until pair accuracy at the start of an iteration nears 50%, then preference optimisation on whatever labelled data exists, so that the preference stage starts from a model that already matches its demonstrations.

Hyperparameters#

Compute and memory#

Training memory equals DPO's: the trainable policy (about 16 bytes per parameter for full fine-tuning with Adam, less with RMSProp's single moment) plus a frozen reference in bf16 (2 bytes per parameter). With LoRA the reference is the base with the adapter disabled, but note the reference changes every iteration, so you merge the previous adapter into the base before starting the next round. Each training step processes two sequences per example (human and generated) through both the policy and the reference, about four forward passes and two backward passes per pair, the same as DPO.

The extra cost is generation: one full response per prompt per iteration. For 50K prompts at 500 tokens that is 25M generated tokens per iteration, which on a modern inference engine is a few GPU-hours for a 7B model. Over four iterations, SPIN costs roughly four DPO runs of the same size plus generation, still well below on-policy RL, which generates many samples per prompt at every step.

Failure modes and how to detect them#

Overshoot and likelihood collapse. Both log-probabilities fall while the margin grows, the same likelihood displacement seen in DPO; at the extreme the model's distribution narrows onto patterns the human data has and its own outputs lacked. Track the human responses' log-probability under the new model; a steady fall means steps are too large. Lower the learning rate or epochs, or raise $\lambda$.

Indistinguishable pairs. As the model improves, its responses resemble the human ones and some pairs become near-duplicates, which give conflicting gradients. Watch the margin at the start of each iteration and the share of pairs whose responses are nearly identical; when the initial reward accuracy hovers near 50%, SPIN has converged and further iterations give nothing.

Copying the data's flaws. SPIN converges to the SFT distribution, including its mistakes, style quirks and length habits. If the SFT data is mediocre, SPIN makes the model a more faithful copy of mediocre data. Audit the data first.

Benchmark-specific gains. The original gains were on leaderboard tasks, with the largest jumps on GSM8K and TruthfulQA. Evaluate on your own task distribution rather than assuming transfer.

Stale opponent bug. Generating $y^{\prime }$ with the model being trained (instead of the frozen previous iteration), or forgetting to update the reference between iterations, silently changes the objective. Log which checkpoint produced each synthetic set.

When to use SPIN#

Minimal implementation#

Because the loss is DPO's, the most robust implementation is a DPO trainer fed with SPIN pairs, with the reference set to the previous iteration. A self-contained PyTorch version of the loss:

import torch
import torch.nn.functional as F

def seq_logprob(model, input_ids, attention_mask, response_mask):
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    lp = torch.gather(logits.log_softmax(-1), 2, input_ids[:, 1:, None]).squeeze(-1)
    return (lp * response_mask[:, 1:]).sum(-1)              # sum over response tokens

def spin_loss(policy, opponent, real, synth, lam=0.1):
    """real / synth: dicts with input_ids, attention_mask, response_mask for (x, y) and (x, y')."""
    with torch.no_grad():
        ref_real, ref_synth = seq_logprob(opponent, **real), seq_logprob(opponent, **synth)
    m = lam * ((seq_logprob(policy, **real) - ref_real) - (seq_logprob(policy, **synth) - ref_synth))
    return F.softplus(-m).mean(), (m > 0).float().mean()    # loss, pair accuracy

With TRL, build a dataset with columns prompt, chosen (the human response) and rejected (the previous iteration's generation), and train with DPOTrainer using beta=0.1 and the previous iteration's checkpoint as ref_model. Repeat for each iteration. The authors' own code, built on the Alignment Handbook, is in the uclaml/SPIN repository. In POFE, SPIN is best treated as an iterated DPO recipe whose rejected column is filled by the previous checkpoint, with the human-response log-probability tracked as the brake.

Further reading#

Chen et al. (2024) for the method, theory and ablations; Rafailov et al. (2023) for the DPO loss it reuses; Tunstall et al. (2023) for the Zephyr models and data used in every SPIN experiment; Yuan et al. (2024) for the self-rewarding variant that replaces the human target with the model's own judgements; Goodfellow et al. (2014) for the adversarial framing.

Frequently asked questions#

References#

1. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu (2024). Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models. arxiv.org/abs/2401.01335
2. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
3. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, et al. (2023). Zephyr: Direct Distillation of LM Alignment. arxiv.org/abs/2310.16944
4. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston (2024). Self-Rewarding Language Models. arxiv.org/abs/2401.10020
5. Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio (2014). Generative Adversarial Networks. arxiv.org/abs/1406.2661
6. UCLA AGI Lab (uclaml) (2024). SPIN official code. github.com/uclaml/SPIN

Related reading