RRHF (Rank Responses to align Human Feedback) explained

RRHF (Rank Responses to align language models with Human Feedback) is an offline alignment method that takes several responses to the same prompt, from any source, scores each with a reward, and trains the model so that its length-normalised log-likelihood ranks the responses in the same order, while also doing supervised fine-tuning on the best one. The loss is a margin-free pairwise hinge over every ordered pair plus cross-entropy. There is no value model, no reference model and, if responses and rewards are precomputed, no generation during training. RRHF belongs to the first wave of "RLHF without RL" methods, alongside SLiC-HF and before DPO, and in practice it behaves as a best-of-n learner: the trained model's reward ends up close to the reward of the best response it was shown. It sits after or in place of SFT, and is a close cousin of rejection-sampling fine-tuning.

Where RRHF came from#

RRHF was proposed in April 2023 by Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang and Fei Huang of Alibaba DAMO Academy and Tsinghua University (Yuan et al., 2023), and published at NeurIPS 2023. The arXiv title adds "without tears", which sums up the motivation. PPO-based RLHF, as in InstructGPT, keeps four models in memory (policy, value, reward, reference), is sensitive to many hyperparameters, and only learns from the policy's own samples. The authors wanted something that adds about 30 lines to an SFT script.

The idea came from summarisation, where BRIO (Liu et al., 2022) and the original SLiC had used ranking losses over sequence probabilities to calibrate generation. RRHF replaces the summarisation metric with a reward: a reward model, human labels, or another LLM acting as judge. And it removes a restriction that PPO has: responses can come from the model itself, from other models (the paper uses ChatGPT, text-davinci-003, LLaMA and Alpaca), or from humans.

On Anthropic's Helpful and Harmless data, with a 6B GPT-J reward model as the proxy judge, RRHF on Alpaca-7B reached an average reward of $-0.96$ (top-p sampling) and $-1.03$ (diverse beam search), against $-1.03$ for PPO on the same model, $-1.18$ for untrained Alpaca and $-1.24$ for the dataset's human-preferred responses. In human evaluation of the diverse-beam variant against PPO, 27 wins, 48 ties and 25 losses: comparable, from a far simpler training loop.

The objective#

For a query $x$, take $k$ responses $y_1,\dots ,y_k$ sampled by any policies ${\rho }_1,\dots ,{\rho }_k$, and score each with a reward function, $r_i=R(x,y_i)$. The model $\pi$ being trained scores each response by its length-normalised conditional log-probability:

The ranking loss penalises every pair whose model scores disagree with their rewards:

The fine-tuning loss is cross-entropy on the highest-reward response $y_{i^{\prime }}$ with $i^{\prime }=\arg {\max }_i{r}_i$:

Four details are deliberate. The sum is unweighted: the authors tried weights of 10 and 100 on the ranking term, as BRIO had suggested, and found them worse. There is no margin: BRIO's margins ${\lambda }_{ij}=(j-i)\lambda$ were dropped because results were good without them and tuning $\lambda$ was expensive. Scores are length-normalised, unlike SLiC-HF's summed log-likelihoods. And there is no KL term, because samples are drawn before training from fixed policies rather than from the moving $\pi$.

What the gradient does

For a violated pair ($r_i<r_j$ but $p_i>p_j$), the hinge's gradient is $\nabla p_i-\nabla p_j$, pushing the worse response's average token log-probability down and the better one's up, each with unit weight on the length-normalised score, so $1/\Vert y\Vert$ per token. A pair already in the right order contributes nothing, even if the scores are tied to the third decimal. Meanwhile $L_{\text{ft}}$ contributes a full SFT gradient, summed over the tokens of the best response, at every step. Two consequences follow. With no margin, the ranking term is satisfied the moment the order is right, so it is a corrector, not a driver. And because the cross-entropy is summed over tokens while each ranking term is an average, $L_{\text{ft}}$ dominates the gradient for responses of any realistic length. RRHF is best understood as SFT on the best of $k$, plus a term that actively pushes down responses the model likes more than better ones.

RRHF next to its neighbours

The comparison with SimPO is instructive: SimPO, a year later, uses the same length-normalised score as RRHF but with a logistic loss and a target reward margin, and without the SFT term. PRO (Song et al., 2023) generalises RRHF's pairwise hinge to a listwise objective over a full ranking.

Worked example: four responses#

A prompt has four responses with rewards and model log-likelihoods:

The reward order is a, d, b, c. The model's order by $p_i$ is b, c, d, a, almost exactly backwards. Go through the six ordered pairs $(i,j)$ with $r_i<r_j$: (d, a) gives $\max (0,-1.1+1.2)=0.1$; (b, a) gives 0.4; (c, a) gives 0.3; (b, d) gives 0.3; (c, d) gives 0.2; and (c, b) gives $\max (0,-0.9+0.8)=0$, because the model already scores b above c. So $L_{\text{rank}}=1.3$. The fine-tuning term is the negative log-likelihood of a, the best response: $L_{\text{ft}}=60$. The total loss is 61.3, of which the ranking term is about 2%.

Length normalisation changes the picture. Ranked by summed log-probability, the model would order the responses b ($-24$), d ($-44$), a ($-60$), c ($-90$), so the 100-token response c would look worst for no reason other than its length. With normalisation, c's per-token score of $-0.9$ is better than a's $-1.2$, and the hinge correctly flags that c is over-scored relative to a. Without normalisation, ranking losses on sequence probabilities systematically favour short responses, since every extra token adds a negative log-probability.

The algorithm#

# Offline: build candidates and rewards (done once, or once per iteration)
for x in prompts:
    cands[x] = (beam_search(policy_init, x, beams=4)          # or diverse beam / top-p
                + dataset_responses(x))                        # e.g. HH chosen + rejected
    rewards[x] = [reward_model(x, y) for y in cands[x]]

# Training: an SFT loop with one extra term
for batch in loader(prompts):                                  # 1 query per GPU, grad accum 8
    loss = 0
    for x in batch:
        ys, r = cands[x], torch.tensor(rewards[x])
        p = torch.stack([policy.mean_token_logprob(x, y) for y in ys])     # (k,)
        worse = r[:, None] < r[None, :]                                    # r_i < r_j
        rank = torch.relu(p[:, None] - p[None, :])[worse].sum()
        ft = -policy.sum_token_logprob(x, ys[int(r.argmax())])
        loss = loss + rank + ft
    (loss / len(batch)).backward(); opt.step(); opt.zero_grad()

Sampling and training are separate, so the training job needs only the policy in memory. The paper's sampling policies are worth knowing because they mattered more than anything else. Each query got four model samples plus the dataset's two human responses: vanilla beam search (BP), diverse beam search with four groups and diversity penalty 1.0 at temperature 0.8 (DP), or top-p sampling at temperature 0.8 (SP). Training only on the two dataset responses (setting P) gave the same reward, $-1.31$, from all three starting models, which the authors took as evidence that sample quality, not the starting checkpoint, determines the result. Removing $L_{\text{rank}}$ made results worse ($-1.14$ versus $-1.03$ with beam sampling), so the ranking term earns its place despite its small share of the loss.

Iterated and online variants

Iterating helps: resample candidates from the trained model and train again (IP-n). Rewards improved from $-1.02$ (IP-1) to $-0.96$ (IP-2) and $-0.94$ (IP-3), and in human evaluation IP-2 beat the single-round model in 10 of 100 comparisons, with 90 ties and no losses. Fully online sampling (resampling from $\pi$ every 32 steps) pushed the proxy reward to 0.34, but perplexity jumped from about 14 to 63.78 and the outputs were reward hacking: "very friendly but meaningless" responses, including fake dialogue turns in which a "Human" praises the answer. Adding a KL penalty of 0.01 to the reward fixed it (reward $-0.86$, perplexity 19.76), at the cost of a reference model, which the authors note is contrary to the method's purpose.

Data

Prompts, $k$ responses per prompt, and a scalar reward or a ranking per response. The paper used the HH dataset's prompts with $k=6$ (four samples plus two dataset responses), truncated to 192 tokens, and for the Wombat model, Alpaca's prompts with responses from four models scored by ChatGPT. Since RRHF only uses the order of the rewards, any judge that can rank works: human labels, a reward model, or an LLM judge.

Hyperparameters#

The only real tuning decision is how to generate candidates. Use the strongest sampling setup you can afford, include high-quality external responses if you have them, and iterate. If you add anything, add a margin: with none, pairs that are ordered correctly by a hair stop contributing, which matters when candidates are near-duplicates from beam search.

Compute and memory#

Training memory is one model with its optimiser state, about 16 bytes per parameter for full fine-tuning with Adam in mixed precision, the same as SFT; there is no reference or value model. Activations are $k$ times larger than SFT's for the same sequence length, since each step runs $k$ responses per query through the model (the paper's 1 query per GPU with $k=6$ at 192 tokens is a direct consequence). Per step:

against DPO's two responses plus a reference forward pass. Sampling is a separate, embarrassingly parallel job: the paper reports 4 to 6 hours on 8 A100 80GB GPUs to sample with beam, diverse beam or top-p, and 4 to 6 hours to train without online sampling, versus about 30 hours with online sampling. The PPO comparison is the paper's headline: one or two models instead of four.

Toy run: RRHF as a best-of-n learner#

We checked the best-of-n claim in a tabular model where we know every reward. 300 prompts, each with a pool of 12 possible responses with hidden rewards drawn from $\mathcal{N}(0,1)$. The policy is a softmax over the pool, initialised randomly (expected reward 0.019). For each prompt, the candidate set has 4 samples from the policy plus two "provided" responses (the pool's second-best and third-worst, playing the role of the dataset's chosen and rejected). The mean reward of a candidate is 0.069; the mean reward of the best candidate per set is 1.299. In a tabular model the score $p_i$ is just $\log \pi (y_i)$. We trained with gradient steps at learning rate 0.1 and compared RRHF, cross-entropy on the best candidate alone, and the ranking loss alone.

The results line up with the paper's observations. The trained policy's reward converges to the best-of-set reward (1.254 against 1.299 after 300 steps), which is the paper's "best-of-n learner" statement in its simplest form; in a separate run trained to convergence (1,500 steps at learning rate 1.0, a different candidate draw) it matched the set best to three decimals, 1.292 against 1.292. The ranking term matters early: after 30 steps RRHF is at 0.769 against 0.568 for cross-entropy alone, because it actively pushes probability off responses the model over-scores. And ranking alone stalls at 0.223 after 10 steps. Without a margin, it stops the instant every candidate set is ordered correctly, which shows why RRHF needs the cross-entropy term to pull the model towards the best response.

Iterating, with 30 steps per round and fresh samples from the current policy, raised the trained policy's reward from 0.761 to 1.045, 1.159 and 1.218 over four rounds, while the mean candidate reward rose from 0.049 to 0.856. That is the IP-n result: each round's candidates are better, so each round's best-of-n target is better.

Failure modes#

Ceiling set by the candidates. RRHF can only learn what is in its candidate sets. If the best of $k$ is mediocre, so is the model. Track the mean and maximum candidate reward per round; if the trained model's reward is already near the maximum, more epochs will not help, but better or more diverse candidates will.

Reward hacking with online sampling. As soon as samples come from the model being trained, a learned reward model is exploitable, exactly as with PPO. The paper's online run found degenerate praise loops within the proxy reward's blind spots. Watch perplexity on a held-out set and read samples; add a KL penalty if you sample online.

Base models with poor samples. Starting from untuned LLaMA gave the worst results ($-1.34$ with diverse beam and perplexity 67), because its own samples were poor ($-1.89$). Start from an instruction-tuned model or include strong external responses.

Ties and near-duplicates. Beam-search candidates are often near-identical. With no margin and tied rewards, those pairs contribute nothing and $k$ is effectively smaller. Deduplicate and prefer diverse beam search or sampling.

Long responses under-weighted in ranking. Length normalisation removes the short-response bias of summed log-probabilities, but it also means a long response's per-token gradient from the ranking term is small. If a long, bad response is over-scored, it is corrected slowly.

When to use RRHF#

Minimal implementation#

import torch

def token_logprobs(model, input_ids, attention_mask, labels_mask):
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    lp = torch.gather(logits.log_softmax(-1), 2, input_ids[:, 1:, None]).squeeze(-1)
    m = labels_mask[:, 1:].float()
    return (lp * m).sum(-1), m.sum(-1).clamp(min=1)          # summed log-prob, length

def rrhf_loss(model, batch, rank_weight=1.0):
    """batch: k responses for ONE query, stacked: input_ids (k, T), attention_mask, labels_mask; rewards (k,)."""
    summed, length = token_logprobs(model, batch["input_ids"], batch["attention_mask"], batch["labels_mask"])
    p = summed / length                                       # length-normalised scores
    r = batch["rewards"]
    worse = (r[:, None] < r[None, :]).float()                 # pair (i, j) with r_i < r_j
    rank = (torch.relu(p[:, None] - p[None, :]) * worse).sum()
    ft = -summed[r.argmax()]                                  # CE on the best response
    return rank_weight * rank + ft, {"rank": rank.item(), "ft": ft.item()}

The authors' reference implementation, a modified Alpaca training script, is in the GanjinZero/RRHF repository. Most preference libraries do not ship RRHF as a named loss, but the function above drops into any SFT trainer that can batch $k$ responses per prompt. In POFE, an RRHF-style run is a preference recipe with a candidate-generation stage, and the run report shows candidate best-of-$k$ reward next to the trained model's, so the ceiling is visible before you spend more epochs.

Further reading#

Yuan et al. (2023) is the primary source; its ablation table on sampling policies is the most useful part. Liu et al. (2022, BRIO) is the summarisation work whose ranking loss RRHF adapted; Zhao et al. (2022, SLiC) and Zhao et al. (2023, SLiC-HF) are the parallel line from Google. Song et al. (2023, PRO) extends the pairwise hinge to a listwise objective. Dong et al. (2023, RAFT), contemporary with RRHF, fine-tunes on the best-reward samples only; the RRHF paper's ranking-loss ablation is the direct comparison. Ouyang et al. (2022) is the InstructGPT pipeline RRHF set out to simplify.

Frequently asked questions#

References#

1. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang (2023). RRHF: Rank Responses to Align Language Models with Human Feedback without tears. arxiv.org/abs/2304.05302
2. Yixin Liu, Pengfei Liu, Dragomir Radev, Graham Neubig (2022). BRIO: Bringing Order to Abstractive Summarization. arxiv.org/abs/2203.16804
3. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, Peter J. Liu (2022). Calibrating Sequence likelihood Improves Conditional Language Generation. arxiv.org/abs/2210.00045
4. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J. Liu (2023). SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arxiv.org/abs/2305.10425
5. Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, Houfeng Wang (2023). Preference Ranking Optimization for Human Alignment. arxiv.org/abs/2306.17492
6. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang (2023). RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment. arxiv.org/abs/2304.06767
7. Long Ouyang et al. (2022). Training language models to follow instructions with human feedback. arxiv.org/abs/2203.02155
8. GanjinZero (2023). RRHF code repository. github.com/GanjinZero/RRHF

Related reading