ORPO (Odds Ratio Preference Optimization), explained

Odds Ratio Preference Optimization (ORPO) folds supervised fine-tuning and preference alignment into a single training stage. Its loss is the ordinary next-token negative log-likelihood on the chosen response plus a weighted penalty that pushes the odds of generating the chosen response above the odds of generating the rejected one. There is no reference model and no separate SFT run: you start from a pretrained base model and a preference dataset, and train once. In the post-training stack ORPO replaces the usual SFT followed by DPO pair of stages, which makes it attractive when compute, memory or pipeline simplicity is the constraint.

The problem: SFT also teaches what you do not want#

Jiwoo Hong, Noah Lee and James Thorne (KAIST AI) introduced ORPO in ORPO: Monolithic Preference Optimization without Reference Model, posted in March 2024. Their starting observation is about SFT itself. Cross-entropy on chosen responses raises the probability of the right tokens, but it does nothing to penalize the wrong ones directly, and because chosen and rejected responses for the same prompt share vocabulary, topic and style, SFT tends to raise the likelihood of rejected responses too. The paper shows this in a pilot study: fine-tuning OPT-350M on only the chosen responses of HH-RLHF, the log-probabilities of chosen and rejected responses rise together, and rejected responses end up about as likely to be generated as chosen ones.

The mechanism is visible in the cross-entropy gradient. For one position with logits $z$, softmax probabilities $p$ and correct token $c$, the loss is $-\log p_c$ and its gradient with respect to logit $z_j$ is $p_j-1[j=c]$. The correct token is pushed up by $1-p_c$; every other token is pushed down in proportion to its current probability, regardless of whether it is a harmless synonym or the first token of exactly the response you want to avoid. Nothing in the loss knows which alternatives are bad. Meanwhile, shared parameters mean that raising "assistant-like" tokens in the chosen response also raises them in the rejected response, which is written in the same register. A preference signal has to come from somewhere else, and ORPO puts it directly into the SFT objective.

That is why the standard recipe needs a second stage. DPO or PPO then pushes the rejected responses back down, at the cost of a second training run and a frozen reference model in memory. ORPO's argument is that a small penalty applied during SFT can do the contrastive job directly: domain adaptation from the NLL term, and a mild push away from disfavored generations from the odds-ratio term, in one pass.

The objective#

Odds of a sequence

ORPO works with the length-normalized likelihood of a response. For a prompt $x$ and response $y$ of $m$ tokens,

so $P_{\theta }(y\mid x)$ is the geometric mean of the per-token probabilities, a number between 0 and 1. The odds of generating $y$ are

Odds of $k$ mean the model is $k$ times more likely to generate $y$ than not to. The odds ratio of chosen $y_w$ over rejected $y_l$ is ${\mathrm{OR}}_{\theta }(y_w,y_l)={\mathrm{odds}}_{\theta }(y_w\mid x)/{\mathrm{odds}}_{\theta }(y_l\mid x)$.

The loss

${\mathcal{L}}_{\text{SFT}}$ is the standard causal language modeling loss on the chosen response (prompt tokens masked). $\lambda$ weights the preference term; the paper used 0.1 to 0.25. Expanding the log odds ratio:

The first bracket is the log probability ratio of the two responses. The second is a correction that grows as either probability approaches 1. Note what is absent: no ${\pi }_{\text{ref}}$ and no $\beta$. The NLL term is what keeps the model anchored, playing the role that the KL penalty plays in DPO.

The gradient

The paper derives ${\nabla }_{\theta }{\mathcal{L}}_{\text{OR}}=\delta (d)\cdot h(d)$ for a pair $d=(x,y_w,y_l)$, with

(up to the overall sign of a descent step). $\delta$ is a penalty weight that goes to 0 once the chosen odds dominate, like DPO's $\sigma (-h)$. $h$ contrasts the two responses, and the $1/(1-P)$ factors amplify the update for whichever response is already likely. That amplification is the practical difference from a plain probability ratio: when the model is confident about a rejected response, the odds-ratio term pushes harder on it.

Why odds and not probabilities?

A natural alternative is $-\log \sigma (\log P_{\theta }(y_w)-\log P_{\theta }(y_l))$, the probability ratio, which is what you get from DPO with the reference removed. The authors argue that in a combined SFT-plus-preference setting, the probability ratio contrasts too sharply and leads to unwarranted suppression of logits for tokens in disfavored responses, which can cause degeneration; they found the odds ratio gives a milder, more stable penalty. The identity above shows when the choice matters:

When both length-normalized probabilities are small, $1-P\approx 1$ and the two are nearly identical. The difference appears when average token probabilities are high, which is the regime of a model that has already fit the domain. We see exactly this in the toy run below.

ORPO next to its siblings

ORPO and CPO are close relatives: both drop the reference and add NLL on the chosen response. ORPO length-normalizes and uses odds; CPO uses summed log-probs and a probability ratio with $\beta$. SimPO shares the length normalization but drops the NLL term and adds a margin, so it still expects an SFT model as its starting point.

Worked example#

A pair where the chosen response has mean token log-probability $-0.9$ and the rejected response $-1.4$.

• $P_w=e^{-0.9}=0.407$, $P_l=e^{-1.4}=0.247$.
• Odds: $0.407/0.593=0.685$ and $0.247/0.753=0.327$.
• $\log \mathrm{OR}=\log (0.685/0.327)=0.739$. The log probability ratio is $-0.9+1.4=0.5$; the odds correction adds 0.239.
• ${\mathcal{L}}_{\text{OR}}=-\log \sigma (0.739)=0.391$.
• ${\mathcal{L}}_{\text{SFT}}=0.9$ (mean token NLL of the chosen response). Total with $\lambda =0.1$: $0.9+0.039=0.939$. With $\lambda =1$: $1.291$.
• Penalty weight $\delta =1/(1+0.685/0.327)=0.323$. Amplification factors $1/(1-P)$: 1.69 for the chosen response, 1.33 for the rejected.

At $\lambda =0.1$ the preference term is a small fraction of the loss, about 4% here. That is by design. The NLL term does most of the work of learning the domain and format; the odds-ratio term tilts it. It also explains why $\lambda$ is the hyperparameter to watch.

A toy run: what SFT does to rejected responses#

To see the mechanism the paper describes, we built the smallest model that has it: a unigram "language model" over a 100-token vocabulary, starting from a uniform base model. Only 20 tokens are in the task domain: 10 neutral tokens used by every response, 5 "good" tokens and 5 "bad" tokens. Chosen responses (500 of them, 20 tokens each) draw 60% neutral, 38% good and 2% bad tokens; rejected responses draw 60% neutral, 10% good, 30% bad. We trained four ways from the same base for 1,500 Adam steps (learning rate 0.01, batch 32): SFT on chosen only, ORPO with $\lambda =0.1$, ORPO with $\lambda =1$, and the probability-ratio variant with $\lambda =1$.

The best achievable chosen log-prob in this setup is −2.778 (the entropy of the chosen-token distribution). Four observations. First, SFT on chosen responses raised the rejected responses' log-probability by almost a nat per token, from −4.61 to −3.64, because rejected responses share the domain's neutral tokens. This is the paper's motivating effect in miniature. Second, SFT only brought the bad tokens down to the 2% they occupy in the chosen data; it has no reason to go lower. Third, ORPO at $\lambda =0.1$ cut bad-token mass by 40% relative to SFT with no measurable cost on chosen (−2.788 against −2.786). Fourth, $\lambda =1$ suppressed bad tokens almost completely but cost 0.08 nats per token on chosen, matching the paper's Appendix E observation that large $\lambda$ lowers chosen log-probs along with rejected ones.

The probability-ratio variant behaved almost identically to ORPO here. That is the identity from the previous section at work: per-token average probabilities in this toy are around 0.06, where odds and probability are nearly the same thing. The paper's argument for odds concerns the regime of high average probabilities, which this toy does not reach; we report the null result rather than tune the toy until it shows a difference.

The algorithm and data#

Input: base model pi_theta, pairs D = {(x, y_w, y_l)}, lambda
for each minibatch:
    # one forward pass over chosen and rejected, concatenated
    tok_lp_w, tok_lp_l = per-token log-probs of y_w, y_l (prompt masked)
    nll       = -mean over tokens(tok_lp_w)                 # standard SFT loss
    lp_w      = mean(tok_lp_w); lp_l = mean(tok_lp_l)       # length-normalized
    log_odds  = (lp_w - lp_l) - (log(1 - exp(lp_w)) - log(1 - exp(lp_l)))
    loss      = nll + lambda * mean(-log_sigmoid(log_odds))
    step optimizer
    log: nll, log_odds, mean sigmoid(log_odds), lp_w, lp_l

Data is the same preference format as DPO: prompt, chosen, rejected. Because ORPO is also doing SFT, the chosen responses must be good enough to imitate: ORPO will learn their style, length and formatting as SFT would. The paper used the HH-RLHF dataset and binarized UltraFeedback, filtering out pairs where chosen equals rejected or either is empty. Its strongest models, Mistral-ORPO-α and Mistral-ORPO-β (7B), were trained on UltraFeedback alone with no separate instruction-tuning set: both were trained for a single epoch on UltraFeedback alone; Mistral-ORPO-α reached 11.33% on AlpacaEval 2.0 and 7.23 on MT-Bench, and Mistral-ORPO-β 12.20% and 7.32, ahead of the Zephyr models trained with SFT then DPO. Phi-2 (2.7B, $\lambda =0.25$) and Llama-2 7B ($\lambda =0.2$) trained with ORPO on UltraFeedback alone exceeded the Llama-2 Chat models on AlpacaEval.

If your chosen responses are weaker than what you want the model to produce, ORPO is the wrong tool: the NLL term will pull the model toward them. A dedicated SFT stage on better demonstrations followed by DPO or SimPO lets the two datasets differ.

Reading an ORPO training run

Log four curves: the NLL on chosen responses, the mean per-token log-prob of chosen and of rejected responses, and the mean log odds ratio. The NLL should fall like an ordinary SFT curve, steeply at first and then flattening, because at small $\lambda$ the SFT term carries most of the gradient. The log odds ratio starts wherever the base model happens to put it, not at zero, since there is no reference to cancel against, and should rise steadily. The rejected log-prob has a characteristic shape: it typically rises early along with the chosen one while the model learns the domain, then flattens or turns down as the odds-ratio term takes effect. In our toy at $\lambda =1$ the turn came at around step 150, where the rejected curve peaked at −3.88 and then fell to −5.22. If the rejected curve never turns, $\lambda$ is too small for your data; if the chosen curve turns down as well, $\lambda$ is too large or the learning rate is too high. The mean of $\log \sigma (\log \mathrm{OR})$, which TRL logs as log_odds_ratio, is the closest analogue to DPO's reward accuracy and saturates quickly, so do not use it to choose a checkpoint.

Hyperparameters#

$\lambda$ trades domain fit for contrast. The paper's ablation (λ in {0.1, 0.5, 1.0}) shows that at 0.1 chosen and rejected log-probs stay close while the rejected ones fall slowly; at 1.0 the odds-ratio term dominates, chosen log-probs fall along with rejected ones, and MT-Bench scores dropped in extraction, math and reasoning relative to 0.1. Start at 0.1 and raise it only if rejected-style outputs persist.

Learning rate follows SFT conventions because the NLL term dominates the gradient. That is the opposite of DPO practice, where rates are an order of magnitude lower. Porting a DPO learning rate into ORPO will usually undertrain the SFT part.

Compute and memory#

ORPO needs one model. Memory for full fine-tuning is roughly 16 bytes per parameter (bf16 weights and gradients, fp32 master weights, Adam moments) plus activations for two sequences per pair, the same as SFT with twice the sequences. There is no reference model, so compared with DPO it saves 2 bytes per parameter of frozen weights and, more importantly, half the forward passes per batch, as the paper points out: two policy passes per pair instead of two policy plus two reference passes. Relative to running SFT on chosen responses and then DPO, the total cost is roughly one SFT-sized run over the preference data with doubled sequence count, versus an SFT run plus a DPO run. With LoRA or QLoRA the savings carry over directly since there is no reference to manage at all.

Failure modes#

λ too high. Chosen log-probs fall with rejected ones; outputs get shorter or terse; reasoning-heavy categories degrade. Watch the chosen mean token log-prob: under ORPO it should rise like an SFT curve, not peak and decline.

Weak chosen data. Because ORPO imitates chosen responses, a preference dataset built for DPO (where the chosen response only needs to beat the rejected one) can teach mediocre style. Check a sample of chosen responses as if they were SFT data, because they are.

No anchor. Without a reference model, nothing explicitly limits drift from the base model's broader capabilities. Run general benchmarks (knowledge, reasoning, safety) alongside the alignment eval, as you would after SFT.

Overfitting across epochs. The paper selected checkpoints by evaluation loss over up to 10 epochs for smaller models. Watch eval NLL on chosen responses and stop when it turns up.

Length effects. The length normalization in the odds term reduces, but does not remove, length bias inherited from the chosen responses via the NLL term. Track output length.

When to use ORPO#

Independent comparisons are mixed. In the SimPO paper's tuned baselines (where, for a fair comparison, ORPO started from the same SFT checkpoints as the other methods, which the authors found worked better than starting from base), ORPO trailed SimPO everywhere and was mixed against DPO: behind on most AlpacaEval 2 numbers, ahead on Arena-Hard in the Mistral-Instruct setting. The KTO paper's ablation found ORPO ($\lambda =0.1$) behind KTO on GSM8K and BBH in its UltraFeedback setup. ORPO's strength is efficiency and simplicity rather than peak benchmark scores. In POFE it is offered as the single-stage option for teams starting from a base checkpoint.

Minimal implementation#

import torch
import torch.nn.functional as F

def log1mexp(x):
    """log(1 - exp(x)) for x < 0, numerically stable."""
    return torch.where(x < -0.693, torch.log1p(-torch.exp(x)), torch.log(-torch.expm1(x)))

def orpo_loss(tok_lp_w, mask_w, tok_lp_l, mask_l, lam=0.1):
    """tok_lp_*: per-token log-probs [B, T]; mask_*: 1 on response tokens."""
    nll = -(tok_lp_w * mask_w).sum() / mask_w.sum()                 # SFT term
    lp_w = (tok_lp_w * mask_w).sum(-1) / mask_w.sum(-1)             # mean log P(y_w)
    lp_l = (tok_lp_l * mask_l).sum(-1) / mask_l.sum(-1)
    log_odds = (lp_w - lp_l) - (log1mexp(lp_w) - log1mexp(lp_l))
    l_or = -F.logsigmoid(log_odds).mean()
    return nll + lam * l_or, {"nll": nll.item(), "log_odds": log_odds.mean().item(),
                              "lp_chosen": lp_w.mean().item(), "lp_rejected": lp_l.mean().item()}

In recent TRL versions ORPO lives under trl.experimental; the weight $\lambda$ is the beta argument:

from trl.experimental.orpo import ORPOConfig, ORPOTrainer

args = ORPOConfig(output_dir="orpo-out", beta=0.1, learning_rate=8e-6,
                  lr_scheduler_type="cosine", warmup_ratio=0.1,
                  num_train_epochs=1, max_length=2048, bf16=True)
trainer = ORPOTrainer(model=base_model, args=args, train_dataset=pairs,
                      processing_class=tokenizer)
trainer.train()

Further reading#

The ORPO paper's appendices hold the gradient derivation and the λ ablation; the others are the closest relatives and comparisons.

• Hong, Lee and Thorne (2024), ORPO: Monolithic Preference Optimization without Reference Model. The method, gradient derivation, λ ablation and odds-versus-probability analysis.
• Rafailov et al. (2023), Direct Preference Optimization. The two-stage baseline ORPO replaces.
• Xu et al. (2024), Contrastive Preference Optimization. A closely related reference-free loss with an NLL term.
• Meng et al. (2024), SimPO. Length-normalized, reference-free, with tuned ORPO baselines.
• Ethayarajh et al. (2024), KTO. Includes an ORPO comparison on UltraFeedback.
• Tunstall et al. (2023), Zephyr. The SFT-then-DPO models ORPO was compared against.

Frequently asked questions#

References#

1. Hong, J., Lee, N., Thorne, J. (2024). ORPO: Monolithic Preference Optimization without Reference Model. arxiv.org/abs/2403.07691
2. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arxiv.org/abs/2305.18290
3. Xu, H., Sharaf, A., Chen, Y., Tan, W., Shen, L., Van Durme, B., Murray, K., Kim, Y. J. (2024). Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation. arxiv.org/abs/2401.08417
4. Meng, Y., Xia, M., Chen, D. (2024). SimPO: Simple Preference Optimization with a Reference-Free Reward. arxiv.org/abs/2405.14734
5. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. arxiv.org/abs/2402.01306
6. Tunstall, L. et al. (2023). Zephyr: Direct Distillation of LM Alignment. arxiv.org/abs/2310.16944
7. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (ORPOTrainer). github.com/huggingface/trl

Related reading