CPO (Contrastive Preference Optimization), explained

Contrastive Preference Optimization (CPO) is a reference-free preference loss: DPO with the reference model replaced by a uniform distribution, plus a negative log-likelihood term on the preferred response. It was designed for machine translation, where the problem is not teaching a model to be helpful but getting it past the quality ceiling of its own training data, and it has since become a general-purpose option for preference tuning without a frozen reference. In the post-training stack it sits after SFT, though its NLL term means it can also do a share of SFT's job.

The problem: imitating references caps quality#

Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray and Young Jin Kim (Johns Hopkins University and Microsoft) introduced CPO in Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation, posted in January 2024 and published at ICML 2024. Their starting point was ALMA, a 13B translation model trained with SFT on small, high-quality parallel data, which was strong but still behind GPT-4 and the best WMT systems.

The diagnosis was about the data. SFT trains a model to imitate reference translations, so the references cap what it can learn. And the references, though written by humans and drawn from the curated FLORES-200 benchmark, were not uniformly better than model output. Scored with reference-free quality-estimation models, ALMA's translations were rated above the human reference for 73.24% of sentences by one metric (KIWI-XXL) and 60.17% by another (XCOMET) in the into-English direction. If the model is often better than its targets, more imitation cannot help. What is needed is a signal that says "this adequate translation is worse than that one", and a way to learn from it cheaply.

Deriving the CPO objective#

Start from DPO:

The authors point out two costs: memory, because two full models must be held, and speed, because the reference must be run on every batch. Both disappear if ${\pi }_{\text{ref}}$ is a uniform prior $U$ over outputs, because then $U(y_w\mid x)=U(y_l\mid x)$ and the two reference terms cancel inside the difference:

Why is that a reasonable approximation rather than a hack? The paper's Theorem 1 says that if the reference is taken to be ${\pi }_w$, an ideal policy that exactly matches the distribution of preferred data, then the DPO loss $\mathcal{L}({\pi }_{\theta };{\pi }_w)$ plus a constant is upper-bounded by $\mathcal{L}({\pi }_{\theta };U)$. Minimizing the uniform-reference loss therefore minimizes an upper bound on DPO against the policy you actually want to reach. This reframes the reference: not "the SFT model we must not drift from" but "the target distribution", which is unknown but drops out of the bound.

Removing the reference also removes the anchor, so the authors add a behavior-cloning constraint that keeps the policy close to the preferred data, ${\mathbb{E}}_{(x,y_w)}[\mathrm{KL}({\pi }_w(y_w\mid x)\Vert {\pi }_{\theta }(y_w\mid x))]<ϵ$, which reduces to an NLL term on preferred responses. The full CPO objective is

In implementations the NLL term usually carries a weight $\alpha$ (TRL's cpo_alpha, default 1.0), giving ${\mathcal{L}}_{\text{prefer}}+\alpha {\mathcal{L}}_{\text{NLL}}$.

The gradient

With $h=\beta (\log {\pi }_{\theta }(y_w)-\log {\pi }_{\theta }(y_l))$,

The chosen response gets the DPO-style push plus a constant NLL push that never vanishes; the rejected response gets only the DPO-style push down. That asymmetry is the point. In DPO both can fall together (likelihood displacement); in CPO the chosen response has a floor under it.

CPO next to its siblings

CPO is very close to SLiC-HF (Zhao et al., 2023) with a logistic loss in place of the hinge, and close to the "DPO plus NLL" recipes used by Iterative RPO and Llama 3. TRL's CPO trainer is also the standard home for SimPO: with loss_type="simpo" and cpo_alpha=0 it is SimPO, and with cpo_alpha>0 it is a SimPO-plus-NLL hybrid often called CPO-SimPO.

The data recipe that made it work#

CPO's results depend as much on how the pairs were built as on the loss. For each source sentence in FLORES-200 (development and test sets, about 2,000 sentences per direction, ten directions between English and Czech, German, Icelandic, Chinese and Russian), the authors collected a triplet: the human reference, a GPT-4 translation and an ALMA-13B-LoRA translation. Each was scored by two reference-free evaluators (KIWI-XXL and XCOMET), the scores averaged, the highest-scoring translation taken as preferred and the lowest as dis-preferred, and the middle one discarded. About 20K pairs, plus 1K human-labeled pairs in two directions.

Two ablations underline the data's role. Removing either the GPT-4 translations or the model's own translations from the triplets hurt, in different directions (into English and out of English). And replacing the natural dis-preferred translations with artificially noised versions of the chosen ones (random word deletions and swaps) gave substantially worse results on every metric. Good negatives are near-misses, the "adequate but not perfect" translations the paper's subtitle refers to, not obviously broken text.

Adapting the recipe outside translation

The ingredients transfer to any task where you have reference outputs of uneven quality and a scorer you trust more than the references. For summarization, the candidate pool can be the human summary, a larger model's summary and two samples from your model, scored by a factuality checker and a judge. For code, the gold solution, a stronger model's solution and your model's attempts, scored by tests first and a style or efficiency judge second. For structured extraction, the annotated output, a stronger model and your own, scored by field-level accuracy against a held-out key. Three rules from the paper carry over. Put your own model's outputs in the pool, because that is where useful negatives come from. Include at least one source that is better than your model, because CPO cannot push past the best candidate it sees. And do not privilege the human reference: let the scorer decide, and expect the reference to lose a meaningful share of the time. If it never loses, your references are already better than anything else in the pool, and plain SFT on them is the simpler choice.

Worked example#

A pair of 20-token translations. Under the policy the chosen one has summed log-probability $-30$ and the rejected one $-34$. Take $\beta =0.1$ and $\alpha =1$.

• A uniform reference over a 32,000-token vocabulary assigns both sequences $20\log (1/32000)\approx -207.5$. With equal lengths the two reference terms cancel exactly in the DPO margin: $h=0.1\times [(-30+207.5)-(-34+207.5)]=0.1\times 4=0.4$, identical to $0.1\times (-30-(-34))$.
• ${\mathcal{L}}_{\text{prefer}}=-\log \sigma (0.4)=0.513$. Gradient weight $\sigma (-0.4)=0.401$.
• ${\mathcal{L}}_{\text{NLL}}=30$ (summed). Total loss 30.5.

The scale mismatch in the last line is real. Summed over a sequence, the NLL term is two orders of magnitude larger than the preference term and its gradient on the chosen response is weighted 1 per token, against $\beta \sigma (-h)=0.04$ per token from the preference term. Implementations typically compute the NLL as a per-token mean (TRL does), which shrinks it by the sequence length and makes $\alpha =1$ a much gentler setting. Check which convention your code uses before copying $\alpha$; it changes the balance between imitation and contrast by a factor of the response length.

Results#

Starting from ALMA-13B-LoRA and training only LoRA weights (rank 16, about 12M parameters) for one epoch on the 22K pairs, with $\beta =0.1$, batch size 128 and maximum length 512, the resulting ALMA-13B-R matched or exceeded GPT-4 and the WMT competition winners on WMT'21, WMT'22 and WMT'23 by reference-free metrics. On WMT'22 out of English it averaged 85.74 KIWI-XXL and 94.05 XCOMET, against GPT-4's 83.83 and 93.23 and the WMT winners' 84.81 and 93.78. A human evaluation of 400 Chinese-to-English sentences preferred ALMA-13B-R over the original ALMA. The loss ablation found that both terms matter: training with ${\mathcal{L}}_{\text{prefer}}$ alone or ${\mathcal{L}}_{\text{NLL}}$ alone (which is just SFT on the preferred translations) was worse than the sum, and adding the NLL term to DPO also improved DPO.

Outside translation the evidence is thinner. In the SimPO paper's tuned chat baselines, CPO (with $\lambda =1$ on the NLL term) was among the weaker methods in several settings, sometimes barely above SFT on AlpacaEval 2, while doing relatively well on Arena-Hard in others. The reasonable reading: CPO is excellent when the chosen responses are worth imitating and negatives are hard near-misses, as in its original setting; for open-ended chat its NLL term can over-anchor to the chosen data.

A toy run#

We rebuilt the paper's setting in miniature. A log-linear policy over 200 candidate responses with 16 shared features and a true quality score (standardized to unit variance). Three "systems" sample responses: the starting model itself (expected quality 0.26), a "human reference" distribution that is clearly better (1.01), and a stronger system (1.42); the best single response scores 2.68. For 3,000 sources we drew one output from each system, scored the triplet with a noisy metric (true quality plus Gaussian noise, standard deviation 0.3), and kept the best as chosen and the worst as rejected (2,957 pairs after dropping ties). Chosen responses came 58% from the strong system, 31% from the human reference and 11% from the model itself; rejected responses came 67% from the model, 23% from the reference and 10% from the strong system. Adam, learning rate 0.02, batch 64, 2,000 steps, $\beta =0.1$, three seeds.

The paper's central story reproduces. SFT on references reaches exactly the references' quality (1.00) and stops. Imitating the best of each triplet instead (NLL on chosen) already does better, and adding the contrastive term improves on that. The weight on the NLL term is a dial between anchoring and movement: as $\alpha$ falls from 1 to 0.02 quality rises from 1.64 to 1.96 and KL doubles. Replacing natural rejected responses with random ones (our stand-in for artificially noised negatives) cost 0.15 at $\alpha =0.02$, matching the paper's finding that near-miss negatives teach more.

Without any NLL term, the preference loss alone travelled furthest, to KL 7.9, and in this toy that bought the most quality, because the true quality is exactly the direction the labels reveal. In a real model that much drift is where fluency and untested capabilities erode, which is why the paper keeps the NLL term and why we would not read this row as a recommendation.

The algorithm#

Input: SFT model pi_theta, sources X, candidate systems S (incl. pi_theta and references),
       quality scorer Q (reference-free metric, reward model or judge), beta, alpha
# 1. build data
for x in X:
    cands = [s(x) for s in S]; scores = [Q(x, c) for c in cands]
    y_w, y_l = cands[argmax(scores)], cands[argmin(scores)]   # drop the middle
# 2. train (usually one epoch)
for each minibatch:
    lw = sum_logp(pi_theta, x, y_w); ll = sum_logp(pi_theta, x, y_l)
    l_prefer = mean(-log_sigmoid(beta * (lw - ll)))
    l_nll    = mean token NLL of y_w
    loss = l_prefer + alpha * l_nll
    step optimizer

Hyperparameters#

$\beta$: 0.1 in the paper (the DPO default); the SimPO authors searched 0.01 to 0.1 for their CPO baselines. Without a reference, $\beta$ is purely a temperature on the summed log-prob difference; because that difference is typically several nats for long sequences, small $\beta$ keeps the sigmoid out of saturation.

$\alpha$ (NLL weight): 1.0 by default with per-token-mean NLL. Lower it if the model stays too close to the chosen data and contrast has little effect; raise it if chosen log-probs fall or general capabilities regress.

Learning rate and adapters. The paper does not state its learning rate; it trained LoRA adapters only, for a single epoch with warmup ratio 0.01. For full fine-tuning, start in the DPO range (5e-7 to 1e-6 at 7B to 13B) and expect LoRA to need roughly ten times higher.

Data. Ensure the candidate pool includes the model's own outputs (so negatives are on-policy near-misses) and at least one stronger source (so positives exceed the model). Use two or more scorers and average them to reduce metric gaming.

Compute and memory#

CPO needs one model. Per pair: forward and backward through the policy on chosen and rejected, about 2x an SFT step on a single response; the NLL term reuses the chosen forward pass. Memory is SFT's on two sequences: about 16 bytes per parameter for full fine-tuning with Adam in mixed precision plus activations, with no frozen reference. That is the saving the paper set out to get: no second model in memory and no sequential reference forward passes. With LoRA, as in the paper, a 13B model trains on modest hardware. The upstream cost to budget is data construction: generating candidates from several systems and scoring every candidate with one or more large evaluator models, which can exceed the training cost itself.

Failure modes#

Metric gaming. If one automatic scorer chooses the pairs and the same scorer evaluates the model, gains can be partly artifacts. The paper used two scorers for selection and reported on three metrics plus human evaluation. Hold out an evaluator you did not train against.

Over-anchoring. With a large $\alpha$ (or a summed NLL), CPO behaves like SFT on chosen responses and the contrast barely registers. Diagnose by comparing to an NLL-only run on the same pairs: if they are indistinguishable, lower $\alpha$.

Easy negatives. Negatives that are obviously broken (truncated, off-language, noised) teach the model what it already knows. Prefer near-misses from your own model.

Length effects. Summed log-probs mean longer chosen responses carry more NLL and more contrast weight. In translation lengths are tied to the source; in open-ended generation, monitor output length or switch to the length-normalized SimPO form.

Metrics to log: ${\mathcal{L}}_{\text{prefer}}$ and ${\mathcal{L}}_{\text{NLL}}$ separately, chosen and rejected log-probs, reward accuracy, output length, and at least one evaluator not used for pair selection.

When to use CPO#

In POFE, the triplet-scoring step is the part teams usually need help with: generating candidates from the current model and a stronger one, scoring them with more than one evaluator, and keeping one evaluator out of the loop for honest measurement.

Minimal implementation#

import torch
import torch.nn.functional as F

def cpo_loss(tok_lp_w, mask_w, tok_lp_l, mask_l, beta=0.1, alpha=1.0):
    """tok_lp_*: per-token log-probs [B, T]; mask_*: 1 on response tokens."""
    lw = (tok_lp_w * mask_w).sum(-1)          # summed log-probs
    ll = (tok_lp_l * mask_l).sum(-1)
    l_prefer = -F.logsigmoid(beta * (lw - ll)).mean()
    l_nll = -(tok_lp_w * mask_w).sum() / mask_w.sum()   # per-token mean NLL on chosen
    return l_prefer + alpha * l_nll, {"l_prefer": l_prefer.item(), "l_nll": l_nll.item(),
                                      "reward_acc": (lw > ll).float().mean().item()}

In TRL (under trl.experimental in recent versions):

from trl.experimental.cpo import CPOConfig, CPOTrainer

args = CPOConfig(output_dir="cpo-out", beta=0.1, cpo_alpha=1.0, loss_type="sigmoid",
                 learning_rate=5e-7, num_train_epochs=1, max_length=512, bf16=True)
trainer = CPOTrainer(model=sft_model, args=args, train_dataset=pairs,
                     processing_class=tokenizer, peft_config=lora_config)
trainer.train()

Further reading#

Sections 3 and 5 of the CPO paper cover the derivation and the ablations; the others are the methods it builds on or competes with.

• Xu et al. (2024), Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. The derivation, Theorem 1, data recipe and ablations.
• Rafailov et al. (2023), Direct Preference Optimization. The loss CPO approximates.
• Hejna et al. (2023), Contrastive Preference Learning. The source of the behavior-cloning regularizer CPO cites.
• Zhao et al. (2023), SLiC-HF. A hinge-loss relative with the same NLL anchor.
• Pang et al. (2024), Iterative Reasoning Preference Optimization. DPO plus NLL in an iterative loop.
• Meng et al. (2024), SimPO. Length-normalized reference-free loss, with tuned CPO baselines.

Frequently asked questions#

References#

1. Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., Kim, Y. J. (2024). Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. arxiv.org/abs/2401.08417
2. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
3. Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., Sadigh, D. (2023). Contrastive Preference Learning: Learning from Human Feedback without RL. arxiv.org/abs/2310.13639
4. Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., Liu, P. J. (2023). SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arxiv.org/abs/2305.10425
5. Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., Weston, J. (2024). Iterative Reasoning Preference Optimization. arxiv.org/abs/2404.19733
6. Llama Team, AI @ Meta (Grattafiori, A. et al.) (2024). The Llama 3 Herd of Models. arxiv.org/abs/2407.21783
7. Meng, Y., Xia, M., Chen, D. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. arxiv.org/abs/2405.14734
8. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (CPOTrainer). github.com/huggingface/trl

Related reading