Self-rewarding language models, explained

Self-rewarding language models generate their own preference data: the model writes several responses to a prompt, then, prompted as a judge, scores those same responses, and the best- and worst-scored responses become a preference pair for DPO. Repeat for a few rounds and both skills are trained at once, since the judge is the model being improved. There is no separate reward model, frozen or otherwise, and after a small seed set no human preference labels. The method sits after SFT as an iterative preference-tuning loop, close to RLAIF (which uses an external model's judgements) and to SPIN (which uses self-play against human demonstrations instead of a judge).

Where it came from#

Self-Rewarding Language Models was published in January 2024 by Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu and Jason Weston at Meta and NYU (Yuan et al., 2024), and appeared at ICML 2024. The motivation is in the first line of the abstract: to build superhuman agents, models will need superhuman feedback. Standard RLHF trains a reward model on human preferences and then freezes it, so the reward model is capped by the quality of human labels and cannot improve while the policy does. If the policy itself provides the reward, and the policy gets better during training, the reward could improve too.

The experiment started from Llama 2 70B. Three iterations of self-rewarding training raised its AlpacaEval 2.0 win rate against GPT-4 Turbo from 9.94% (iteration 1, the SFT model) to 15.38% and then 20.44%, ahead of Claude 2 (17.19%), Gemini Pro (16.85%) and GPT-4 0613 (15.76%) on the leaderboard of the time. Its ability as a judge also improved without any new judging data: pairwise agreement with human rankings went from 78.7% to 80.4% to 81.7%. A follow-up from an overlapping Meta team, Meta-Rewarding (Wu et al., 2024), added a third role in which the model judges its own judgements, and took Llama-3-8B-Instruct from 22.9% to 39.4% length-controlled win rate on AlpacaEval 2.

The method, piece by piece#

Seed data and the first model

Start from a pretrained base model $M_0$ and two small human datasets. IFT (instruction fine-tuning) data are prompt and response pairs: the paper used 3,200 high-quality first-turn examples from Open Assistant. EFT (evaluation fine-tuning) data teach the model to act as a judge: a prompt asking it to score a response, and a target consisting of a short justification followed by a score out of 5. The paper built 1,630 training and 541 evaluation examples from Open Assistant's human-ranked responses, keeping only generated justifications whose scores agreed with the human ranking and downsampling the over-represented score of 4. $M_1$ is $M_0$ fine-tuned with SFT on IFT plus EFT.

The judge prompt matters more than one might expect. The paper's prompt awards points additively (one point each for relevance, substantial coverage, usefulness, clear AI-assistant perspective and expert quality). An alternative prompt from earlier work, which describes the five scores as quality buckets, gave 26.6% pairwise agreement with humans on the SFT baseline, against 65.1% for the additive prompt.

Self-instruction creation

Each iteration builds new training data in three steps. Generate new prompts by few-shot prompting from the seed IFT prompts (in the main experiments a fixed Llama 2-Chat 70B did this, with 8-shot prompts and the usual Self-Instruct filters). Sample $N=4$ candidate responses per prompt from the current model $M_t$ at temperature 0.7 and top-p 0.9. Score each candidate with $M_t$ itself using the judge prompt, three times with sampling, and average:

Pair construction and training

For prompt $x_i$ with candidates $y_i^1,\dots ,y_i^N$, take the highest and lowest scored, and drop the prompt if they tie:

This set is AIFT($M_t$), AI-feedback training data. $M_{t+1}$ is $M_t$ trained with DPO on it. The paper does not spell out the reference policy; the standard iterative-DPO choice, assumed here, is $M_t$ itself:

The paper's sequence is $M_1$ (SFT on IFT and EFT), $M_2$ (DPO on AIFT($M_1$), 3,964 pairs) and $M_3$ (DPO on AIFT($M_2$), 6,942 pairs). The pair counts are small: the whole method is driven by about eleven thousand self-labelled pairs.

Why it can work, and what bounds it

Think of the round's pair data as preferences labelled by a noisy judge with accuracy $a_t$, the probability that the chosen response is really the better one. DPO on pairs with label accuracy $a_t$ learns, roughly, a reward whose margins shrink towards zero as $a_t\to \frac{1}{2}$: label noise acts like an implicit temperature. Self-rewarding bets on two things. First, that $a_t$ is well above $\frac{1}{2}$, which the additive prompt and score averaging are designed for. Second, that $a_{t+1}\ge a_t$: training on instruction following transfers to judging, so the next round's labels are at least as good. The paper's measurements support both for two rounds (agreement 78.7%, 80.4%, 81.7%), and its own discussion notes that the effect "likely saturates in real-world settings". Meta-Rewarding was motivated precisely by that saturation: if the judge stops improving, the actor quickly overfits its judge.

There is also a ceiling on what any round can teach. The chosen response is one of the model's own $N$ samples, so each round can at best move the model towards its own best-of-$N$. Like rejection sampling, the method sharpens what the model can already produce more than it adds new capability. The paper's breakdown is consistent with that: gains were large in most instruction categories but small on mathematics and logical reasoning, where MT-Bench went from 3.93 (SFT) to only 4.17 ($M_3$).

Worked example: one prompt through a round#

The model writes four responses, and judges each three times:

The pair is $(y^1,y^4)$. With a single judgement per response, $y^1$ and $y^3$ would have tied at 4 on the first draw, and if the lowest draw had also been 4 the prompt would have been discarded. Averaging three draws turns a coarse 6-level scale into a 16-level one and breaks most ties. At the start of DPO both log-ratios are 0, so the loss is $-\log \sigma (0)=\log 2=0.693$ and the gradient pushes $\log \pi (y^1)$ up and $\log \pi (y^4)$ down with weight $\beta \sigma (0)=0.1\times 0.5=0.05$ each. Note what the pair does not contain: $y^3$, which the judge rated nearly as good as $y^1$, and the information that $y^1$ won by 1.67 points rather than 0.33. Self-rewarding uses the judge's scores only to pick a pair.

Toy run: judge noise, averaging and length bias#

We simulated the pair-construction step to see how much the averaging and tie-dropping buy. 20,000 prompts with 4 responses each. Response quality is a prompt-level term $\mathcal{N}(0,0.7^2)$ plus a within-prompt term $\mathcal{N}(0,0.4^2)$, so a model's own samples are similar, as they are in practice. The judge outputs $\mathrm{round}(3.5+q+b\cdot \ell +\epsilon )$ clipped to 0 to 5, where $\ell$ is the response's standardised length, $b$ a length bias, and $\epsilon \sim \mathcal{N}(0,{\sigma }^2)$ judge noise. We measure how often the chosen response really is the better of the pair.

Averaging is worth roughly ten points of label accuracy (81.9% to 91.9% at moderate noise, 69.7% to 80.5% at high noise) and almost eliminates discarded prompts. A modest length bias costs a few points of accuracy, but its main effect is systematic: the chosen response is 0.8 to 1.0 standard deviations longer than the rejected one. Averaging makes that worse, not better, because it removes noise but not bias, so the bias decides more of the close calls.

To see what that does over iterations, we ran four rounds of self-judged iterative DPO in a tabular model: 2,000 prompts, a pool of 12 responses each, 4 samples per round from the current policy, the judge above with $\sigma =1$ and three averaged judgements, and DPO with $\beta =0.1$ from the current model each round. Three seeds.

With an unbiased judge, true quality rose every round (0.007, 0.075, 0.134, 0.185, 0.226) and length did not move. With a length bias of 0.3, quality rose slightly less (to 0.205 after four rounds, about 9% lower) while mean length climbed steadily, to +0.37 standard deviations. The paper saw the same pattern at full scale: average AlpacaEval response length went from 1,092 for $M_1$ to 1,552 for $M_2$ and 2,552 for $M_3$. The authors flag length as a possible factor in the win-rate gains. Meta-Rewarding responds by choosing the shorter of two responses when their scores are close.

The algorithm#

M = sft(base_model, ift_seed + eft_seed)                 # M1: follow instructions AND judge
for t in range(2):                                       # paper: two DPO rounds (M2, M3)
    prompts = self_instruct(fixed_prompt_model, ift_seed, n=...)   # new prompts, filtered
    pairs = []
    for x in prompts:
        ys = [M.generate(x, temperature=0.7, top_p=0.9) for _ in range(4)]
        scores = [mean(parse_score(M.generate(judge_prompt(x, y), temperature=0.7, top_p=0.9))
                       for _ in range(3)) for y in ys]
        if max(scores) > min(scores):
            pairs.append((x, ys[argmax(scores)], ys[argmin(scores)]))
    M = dpo(init=M, ref=M, data=pairs, beta=0.1, lr=1e-6, lr_end=1e-7, batch_size=16)
    # early stopping: checkpoint every 200 steps, compare generations against the previous checkpoint

Two parts of the loop are easy to get wrong. Score parsing: the judge's output must end in a parseable "Score: k"; count parse failures and treat them as missing, not zero. And early stopping: the paper saved a checkpoint every 200 DPO steps and evaluated generations pairwise against the previous checkpoint on 253 validation prompts with an external judge (Claude 2). Without an independent check, the only signal that the round is going well is the model's own opinion.

Data

A few thousand human instruction-following examples, a couple of thousand judging examples (or a judge prompt that already works zero-shot on your base model), and a source of prompts. The paper's Appendix A.3 reports that IFT data alone already gives a usable judge, but that EFT data improves agreement with humans markedly (65.1% to 78.7%). The prompt set should cover the skills you care about: the paper attributes its weak maths and reasoning gains partly to Open Assistant prompts under-representing them.

Hyperparameters#

The decisive choice is not a number but the judge prompt and the decision rule for pairs. Validate the judge on a held-out set of human-ranked responses before every round (pairwise accuracy, Spearman correlation with human ranks) and log the score histogram: if most responses get the same score, pairs will be few and noisy.

Compute and memory#

Each round has three costs. Generation: $N$ responses per prompt. Judging: $N\times$ (number of judgements) generations of a justification of up to 100 words, so 12 judge generations per prompt in the paper's setting, which typically costs more than generating the responses themselves. Training: one DPO pass, which needs the policy plus a frozen reference, about $16N_{\text{params}}+2N_{\text{params}}$ bytes with Adam in mixed precision (or precompute the reference log-probabilities and skip the second copy).

with $J$ judgements per response. The paper did this at 70B; nothing in the method needs one model per role, so memory never exceeds that of DPO, and there is no reward model to train or serve. Compared with PPO-RLHF it is far cheaper per round; compared with offline DPO on an existing dataset it adds the generation and judging passes.

Failure modes#

Length inflation. The most consistent failure, seen in the paper and reproduced in our toy. Track mean response length per round and report length-controlled win rates (Dubois et al., 2024); consider a length-aware decision rule (Meta-Rewarding prefers the shorter response when scores are close) or a length penalty in the score.

Judge saturation and self-bias. If the judge stops improving, each round's labels are no better than the last and the model starts fitting its own quirks. Evaluate the judge on held-out human rankings every round; stop or add a meta-judge step when agreement plateaus.

Score collapse. When the model's responses converge in quality, the judge gives everything the same score, ties proliferate and pairs dwindle. A rising discard rate is the symptom. More diverse sampling, more candidates or a finer score scale help.

Capability regression. On NLP benchmarks the paper's models mostly held steady, but some dropped (ARC-Challenge from 57.51 for $M_1$ to 53.13 for $M_3$, Natural Questions from 35.48 to 31.86). Run a fixed regression suite after each round.

Domains the judge cannot assess. A model judging its own maths answers without a verifier mostly rewards the look of a good solution. For checkable tasks, use a verifier, as in RLVR or Iterative RPO (Pang et al., 2024), which builds pairs from correct and incorrect chains of thought.

Evaluation circularity. If the model, the judge and the early-stopping signal are all the same model, everything can look good while nothing is. The paper used GPT-4 for evaluation and Claude 2 for checkpoint selection; keep at least one independent judge in the loop.

When to use self-rewarding training#

Minimal implementation#

import re, statistics

JUDGE = """Review the user's question and the corresponding response using the additive 5-point
scoring system described below. ... (relevance, coverage, usefulness, AI-assistant clarity, expertise)
User: {prompt}
<response>{response}</response>
After examining the user's instruction and the response:
- Briefly justify your total score, up to 100 words.
- Conclude with the score using the format: "Score: <total points>"
"""

def judge(generate, prompt, response, n=3):
    scores = []
    for _ in range(n):
        out = generate(JUDGE.format(prompt=prompt, response=response), temperature=0.7, top_p=0.9)
        m = re.search(r"Score:\s*([0-5])", out)
        if m:
            scores.append(int(m.group(1)))
    return statistics.mean(scores) if scores else None       # None: parse failure, skip

def build_pairs(generate, prompts, n_candidates=4, n_judgements=3):
    pairs, discarded = [], 0
    for x in prompts:
        ys = [generate(x, temperature=0.7, top_p=0.9) for _ in range(n_candidates)]
        scored = [(judge(generate, x, y, n_judgements), y) for y in ys]
        scored = [(s, y) for s, y in scored if s is not None]
        if len(scored) < 2 or max(scored)[0] == min(scored)[0]:
            discarded += 1; continue
        pairs.append({"prompt": x, "chosen": max(scored)[1], "rejected": min(scored)[1]})
    return pairs, discarded

# Each round: pairs, _ = build_pairs(model_generate, new_prompts)
# then train with any DPO implementation, e.g. TRL's DPOTrainer(model=M_t, ref_model=None, beta=0.1)

The pairs feed any DPO trainer; TRL's DPOTrainer with its default sigmoid loss and $\beta =0.1$ matches the paper's training step. Ties are broken arbitrarily by max here; with averaged scores they are rare. In POFE, self-rewarding is an iterative recipe in which the judge prompt is versioned, judge agreement on a held-out ranked set is reported before each round, and response length is plotted across rounds next to win rate.

Further reading#

Yuan et al. (2024) is the primary source; its Table 4 on judge quality across iterations is the key evidence. Wu et al. (2024) for Meta-Rewarding, which addresses judge saturation and length bias. Zheng et al. (2023) for LLM-as-a-Judge and MT-Bench, the evaluation tool the method turns into a training signal. Pang et al. (2024) for Iterative RPO, the same group's iterative DPO for reasoning with verified answers. Rafailov et al. (2023) for DPO itself.

Frequently asked questions#

References#

1. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, Jason Weston (2024). Self-Rewarding Language Models. arxiv.org/abs/2401.10020
2. Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, Sainbayar Sukhbaatar (2024). Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge. arxiv.org/abs/2407.19594
3. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arxiv.org/abs/2306.05685
4. Richard Yuanzhe Pang, Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, Jason Weston (2024). Iterative Reasoning Preference Optimization. arxiv.org/abs/2404.19733
5. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
6. Yann Dubois, Balázs Galambosi, Percy Liang, Tatsunori B. Hashimoto (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arxiv.org/abs/2404.04475
7. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (DPOTrainer). github.com/huggingface/trl

Related reading