SLiC-HF (Sequence Likelihood Calibration with Human Feedback) explained

SLiC-HF (Sequence Likelihood Calibration with Human Feedback) aligns a language model to human preferences with a ranking loss on whole-sequence log-likelihoods: for each preference pair, push the log-probability of the preferred response above that of the rejected one by a fixed margin, and add a cross-entropy term that keeps the model close to its supervised behaviour. There is no reward-model-in-the-loop, no sampling during training and no value network. It is one of the first offline alternatives to PPO-based RLHF, posted twelve days before DPO, and its hinge loss survives today as one member of the family of pairwise preference losses (DPO's logistic, IPO's squared, SLiC's hinge). It runs after SFT, on preference pairs that are either collected from humans or produced by ranking the SFT model's own samples.

Origin: from calibration to human feedback#

SLiC began as a fix for a decoding problem. In Calibrating Sequence Likelihood Improves Conditional Language Generation, Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh and Peter J. Liu at Google (Zhao et al., 2022) observed that models trained by maximum likelihood assign sequence probabilities that do not track sequence quality: beam search returns high-likelihood outputs that are not the best outputs. Their remedy was a third stage after pre-training and fine-tuning. Decode $m$ candidates from the fine-tuned model, rank them by similarity to the reference, and train the model so its sequence likelihoods respect that ranking. They compared four calibration losses (rank, margin, list-wise rank and expected reward) and two regularisers (cross-entropy and KL to the fine-tuned model). The rank loss worked best, the two regularisers performed similarly, and about 85% of the gain remained even without regularisation.

In May 2023 the same group (Zhao, Joshi, Tianqi Liu, Khalman, Saleh and Liu, now at Google DeepMind and Google Research) swapped the similarity-to-reference ranking for human preference and called the result SLiC-HF (Zhao et al., 2023). They tested it on Reddit TL;DR summarisation, using the human feedback collected by Stiennon et al. (2020): 117K SFT examples and 64K preference comparisons of summaries from other models. A T5-Large (770M) policy trained with SLiC-HF was preferred by human raters over the 6B decoder-only RLHF-PPO model of Stiennon et al. 66% to 34% when pairs were labelled by a ranking model, and 56% to 44% when labelled by a pointwise reward model. In a four-way human comparison it was chosen as the best summary 73% of the time, against 13% for the human reference.

The objective#

Let $P_{\theta }(y\mid x)={\prod }_t{P}_{\theta }(y_t\mid x,y_{<t})$ be the model's sequence likelihood. For a preference triple $(x,y^{+},y^{-})$ and a regularisation target $y_{\text{ref}}$, the SLiC-HF loss is

The symbols: $\delta$ is the margin, the gap in nats that the preferred response's log-likelihood must exceed the rejected one's by before the pair stops contributing (1.0 in the paper's experiments); $\lambda$ weights the regulariser; $y_{\text{ref}}$ is either the SFT target for $x$ or the best-ranked of the model's own decodes. Write the log-likelihood margin as $m=\log P_{\theta }(y^{+}\mid x)-\log P_{\theta }(y^{-}\mid x)$. The calibration term is a hinge in $m$, so its gradient is

Every pair that violates the margin gets the same unit push, up on the preferred sequence and down on the rejected one, however badly it violates it. Every pair that satisfies it gets nothing. The regulariser's gradient, $-\lambda \nabla \log P_{\theta }(y_{\text{ref}}\mid x)$, is ordinary SFT on the target.

The regulariser plays the role of PPO's KL penalty but is cheaper: it needs no frozen copy of the SFT weights. The original SLiC paper found a KL regulariser, ${\sum }_t{P}_{\theta }\log (P_{\theta }/P_{{\theta }_{\text{ft}}})$ on the target tokens, worked about as well as cross-entropy.

SLiC next to DPO and IPO

The three classic offline losses differ in two choices: what margin they measure, and what function of the margin they penalise. Tang et al. (2024) showed that, in their reference-normalised forms, all of them are instances of one template, a convex function $f$ applied to a scaled log-ratio margin, which they call Generalized Preference Optimization:

Two differences matter in practice. SLiC-HF's margin is measured on raw log-likelihoods, not on log-ratios to a reference, so it does not need a reference model at all; the price is that the starting point matters, because a pair that the SFT model already ranks correctly by more than $\delta$ is ignored from the first step. And the hinge stops: a won pair exerts no force, so training has a fixed point once every pair clears the margin or is contradicted by another pair. DPO's logistic loss keeps pushing every pair for as long as you train, which is one route to the likelihood displacement discussed in the DPO article. The Statistical Rejection Sampling paper (Liu et al., 2023), by the same Google group, introduced the reference-normalised hinge that TRL implements as loss_type="hinge".

Worked example: two pairs, by hand#

Pair A: the model gives the preferred summary $\log P(y^{+})=-42.0$ and the rejected one $\log P(y^{-})=-41.5$. The margin is $m=-0.5$, the wrong sign. With $\delta =1$ the hinge is $\max (0,1-(-0.5))=1.5$, active, and the gradient adds $-\nabla \log P(y^{+})+\nabla \log P(y^{-})$, unit weight on each sequence.

Pair B: $\log P(y^{+})=-35.0$, $\log P(y^{-})=-37.2$. The margin is $2.2\ge 1$, so the hinge is 0 and the pair contributes no gradient at all.

Now DPO with $\beta =0.1$ on pair B, with reference log-probabilities $-35.5$ and $-37.0$. The log-ratio difference is $\Delta =(-35.0+35.5)-(-37.2+37.0)=0.7$, the loss $-\log \sigma (0.07)=0.659$, and the gradient weight $\beta \sigma (-\beta \Delta )=0.1\times 0.4825=0.048$. DPO is still pushing a pair the policy already ranks correctly, and it will keep pushing it until $\Delta$ is many multiples of $1/\beta$. For pair A (references $-41.0$ and $-41.8$), $\Delta =-1.3$ and the weight is 0.053. Note the scale difference too: SLiC's per-pair gradient weight is 1 while DPO's is at most $\beta$, which is one reason SLiC-HF and DPO use learning rates an order of magnitude apart.

The regulariser is on a different scale again. If $y_{\text{ref}}$ has 30 tokens with $\log P=-30$, the cross-entropy term with $\lambda =0.5$ contributes 15 to the loss and a gradient summed over 30 tokens, while the hinge contributes at most a few nats. Because $\lambda$ balances a token-summed term against a sequence-level hinge, its right value depends on target length; the paper does not report the value it used.

The algorithm#

SLiC-HF comes in two forms, shown in Figure 2. In SLiC-HF-sample-rank, you decode $m$ candidates per training prompt from the SFT model (the paper used 8, temperature 0.7, top-k 40), label pairs with a model trained on the human preferences, and calibrate on those pairs. In SLiC-HF-direct, you calibrate straight on the human preference pairs, which were written for other models. Direct is simpler, essentially fine-tuning on a different loss; sample-rank is better and more stable.

# SLiC-HF-sample-rank
ranker = train_pairwise_ranker(human_pairs)            # "[CONTEXT] .. [SUMMARY A] .. [SUMMARY B]" -> "A" / "B"
pairs = []
for x in sft_prompts:
    cands = sft_model.sample(x, n=8, temperature=0.7, top_k=40)
    pairs += tournament_pairs(ranker, x, cands)        # m - 1 = 7 (winner, loser) pairs per prompt
    y_ref[x] = tournament_winner(ranker, x, cands)     # or the SFT target

model = copy(sft_model)                                # no reference copy needed
for batch in loader(pairs):
    lp_pos = model.seq_logprob(batch.x, batch.y_pos)   # sum of token log-probs
    lp_neg = model.seq_logprob(batch.x, batch.y_neg)
    cal = relu(delta - lp_pos + lp_neg).mean()
    reg = -model.seq_logprob(batch.x, y_ref[batch.x]).mean()
    (cal + lam * reg).backward(); opt.step(); opt.zero_grad()

Pair labelling uses a tournament: with four candidates, compare $c_1$ with $c_2$ and $c_3$ with $c_4$, then the two winners, giving $m-1$ pairs from $m-1$ ranker calls. Decoding and ranking happen once, offline, over the whole training set, which is what makes the method cheap. The paper also trained a pointwise reward model (outputs "Good" or "Bad") as an alternative labeller. The pairwise ranking model was more accurate on held-out human preferences (73.23% versus 71.34%), and SLiC-HF with ranker labels did better: 86.21% win rate against references as judged by the ranker, versus 82.42% with reward-model labels.

Data

You need an SFT set (for the regulariser and the decoding prompts) and preference data to train the ranker or to use directly. The paper used 117K SFT prompts and 64K human comparisons. Increasing decodes per prompt from 8 to 64 barely helped (86.21% to 86.41%); scaling the policy from 770M to 11B helped a lot (to 96.10%). For the regulariser, using the SFT targets or the best-ranked decode made little difference, which means sample-rank works even with no references at all.

Hyperparameters#

Because the hinge is measured on summed log-likelihoods, $\delta$ interacts with length: a 1-nat gap is easy to create on a 300-token response (0.003 nats per token) and harder to reason about across responses of very different lengths. If your pairs vary a lot in length, consider normalising log-likelihoods by length (as RRHF and SimPO do) or measuring margins against a reference model (RSO's hinge-norm).

Compute and memory#

SLiC-HF's training stage is SFT-shaped. Per pair it needs a forward and backward pass over $y^{+}$, $y^{-}$ and $y_{\text{ref}}$, with only the policy in memory: no reference copy (the cross-entropy regulariser replaces the KL), no reward model, no critic. The paper's comparison with PPO, for a policy of $p$ parameters:

With Adam in mixed precision that is roughly 16 bytes per parameter, the same as SFT, plus activations for three sequences per example. DPO needs an extra 2 bytes per parameter for the frozen reference (or precomputed reference log-probabilities). The offline decoding and ranking cost is $m$ generations plus $m-1$ ranker calls per prompt, paid once.

Toy run: the margin is a stopping rule#

We trained a tabular policy to see the hinge's fixed point. 200 prompts, 8 candidate responses each, starting logits drawn from $\mathcal{N}(0,1)$ (the "SFT model") and a hidden quality per response from $\mathcal{N}(0,1)$. For each prompt we sample 4 pairs from the starting policy and label them with a noisy Bradley-Terry judge, $P(a\succ b)=\sigma (2(q_a-q_b))$, giving 800 pairs of which only 53.0% are ranked correctly by the starting policy. We trained with full-batch gradient descent for 6,000 steps, SLiC at learning rate 2 and DPO ($\beta =0.1$) at 20 to match per-pair step sizes, and tracked expected quality ${\mathbb{E}}_{\pi }[q]$ (starting at $-0.058$; the best possible is 1.406) and KL from the starting policy.

SLiC stops. With $\delta =1$, KL reaches 0.288 by step 1,000 and 0.298 by step 3,000, then nothing changes: 94.3% of pairs clear the margin and the remainder are contradicted by pairs with the opposite label, whose pushes cancel. The margin sets where it stops, roughly linearly: KL of 0.17, 0.30, 0.64 and 1.17 for $\delta$ of 0.5, 1, 2 and 4. DPO has no such point. Its KL kept rising, to 1.43 at step 6,000, and the log-probability of the chosen responses, after rising to +0.50 at step 1,000, fell to $-0.95$: mass moved off both responses in each pair, the likelihood displacement familiar from DPO. On the quality-versus-KL frontier the two methods are close (SLiC at $\delta =4$: KL 1.17, quality 0.559; DPO at step 3,000: KL 1.19, quality 0.574). The difference is that SLiC's answer does not depend on when you stop training.

We also ran SLiC with the cross-entropy regulariser ($\lambda =0.1$) on a best-ranked target per prompt. In this tabular model the regulariser behaves like SFT on the targets and never saturates: quality kept rising (0.952 at step 6,000) along with KL (1.42). That is a property of the toy, where one target per prompt can absorb all the probability, more than a prediction about LLMs, but it is a reminder that the cross-entropy term is not a trust region. It pulls towards $y_{\text{ref}}$, not towards the SFT model.

Failure modes#

Length growth with direct calibration. The paper saw it: with SLiC-HF-direct, calibration loss fell as expected but average length kept increasing and did not converge, which the authors attributed to the off-policy human-feedback responses. Sample-rank converged. Watch mean response length per checkpoint, not just loss, and select checkpoints with the ranker.

Pairs ignored from step one. Without a reference, any pair the SFT model already ranks correctly by more than $\delta$ contributes nothing. If most of your pairs are "easy" for the SFT model, the effective dataset is much smaller than it looks. Log the fraction of active pairs at the start and over training.

Ranker bias becomes model bias. Sample-rank learns whatever the ranker prefers. SLiC-HF summaries were longer than RLHF-PPO's (about 37 versus 33 words), with a length-controlled win rate similar to PPO's. Run a length-controlled evaluation alongside raw win rate.

Mis-scaled regulariser. Too small and the model drifts from fluent text, which shows as rising perplexity on held-out SFT targets; too large and calibration has no effect. Track both the hinge's active fraction and SFT perplexity.

Weak ranker. The paper found smaller T5 ranking and reward models did not converge reliably and used an 11B ranker for a 770M policy. A ranker barely above chance turns calibration into noise injection.

When to use SLiC-HF, and what instead#

Minimal implementation#

import torch
import torch.nn.functional as F

def seq_logprob(model, input_ids, attention_mask, completion_mask):
    logits = model(input_ids=input_ids, attention_mask=attention_mask).logits[:, :-1]
    tok = torch.gather(logits.log_softmax(-1), 2, input_ids[:, 1:, None]).squeeze(-1)
    return (tok * completion_mask[:, 1:]).sum(-1)              # summed, not averaged

def slic_hf_loss(model, pos, neg, ref, delta=1.0, lam=0.5):
    """pos, neg, ref: dicts with input_ids, attention_mask, completion_mask."""
    lp_pos = seq_logprob(model, **pos)
    lp_neg = seq_logprob(model, **neg)
    cal = F.relu(delta - lp_pos + lp_neg)
    reg = -seq_logprob(model, **ref) / ref["completion_mask"].sum(-1).clamp(min=1)  # per-token CE
    stats = {"active_frac": (lp_pos - lp_neg < delta).float().mean().item()}
    return (cal + lam * reg).mean(), stats

This version normalises the regulariser per token, which makes $\lambda$ independent of target length; the paper's equation uses the summed log-likelihood. In TRL, DPOTrainer with loss_type="hinge" implements the reference-normalised hinge $\max (0,1-\beta \Delta )$ from RSO, where the margin in log-ratio units is $1/\beta$; you can add an NLL term on the chosen responses by combining loss types (loss_type=["hinge", "sft"] with loss_weights), which approximates SLiC-HF's regulariser with $y_{\text{ref}}=y^{+}$. In POFE, the hinge is one of the loss options on the preference-tuning recipe, and the run page reports the active-pair fraction so you can see when training has reached its fixed point.

Further reading#

Zhao et al. (2023) is the SLiC-HF paper; Zhao et al. (2022) is the original calibration paper with the loss and regulariser ablations. Liu et al. (2023, RSO) combines SLiC-style hinge losses with a reference and better pair sampling. Rafailov et al. (2023) is DPO, published the same month. Tang et al. (2024) is the cleanest unification of DPO, IPO and SLiC as convex losses on a margin. Stiennon et al. (2020) is the source of the TL;DR preference data and the PPO baseline SLiC-HF compared against.

Frequently asked questions#

References#

1. Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, Peter J. Liu (2023). SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arxiv.org/abs/2305.10425
2. Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, Peter J. Liu (2022). Calibrating Sequence likelihood Improves Conditional Language Generation. arxiv.org/abs/2210.00045
3. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu (2023). Statistical Rejection Sampling Improves Preference Optimization. arxiv.org/abs/2309.06657
4. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
5. Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, Bilal Piot (2024). Generalized Preference Optimization: A Unified Approach to Offline Alignment. arxiv.org/abs/2402.05749
6. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, Rémi Munos (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arxiv.org/abs/2310.12036
7. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, Paul Christiano (2020). Learning to summarize from human feedback. arxiv.org/abs/2009.01325
8. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (DPOTrainer, hinge loss). github.com/huggingface/trl

Related reading