Online and iterative DPO, explained

Online and iterative DPO keep the DPO loss but change where the preference pairs come from. Instead of training once on a fixed dataset collected from some other model, you repeatedly sample responses from the current policy, have an annotator (a reward model, an LLM judge, a verifier or people) pick the better one, and train on those fresh pairs. "Online" usually means new pairs every training step; "iterative" means rounds: generate and label a batch of pairs, train for a while, regenerate with the improved model, repeat. Both sit in the same slot as DPO, after SFT, but they behave much more like RLHF with PPO: on-policy data, an external reward signal, and a training loop that includes generation. Most strong open post-training recipes since 2024 use some version of this.

The problem: offline DPO trains on someone else's mistakes#

Standard DPO is offline and usually off-policy. The pairs were sampled before training began, often from different models, and the policy never sees what its own outputs look like to the annotator. Three consequences follow.

First, coverage. The loss can only shape probability on responses that appear in the data. As training moves the policy, it starts generating responses that no pair ever compared, and nothing corrects them. Xu et al. (2024) show theoretically that DPO can find solutions that exploit such out-of-distribution responses, and found well-tuned PPO ahead of DPO across dialogue and code benchmarks. Second, signal decay. Once the policy separates the training pairs, their gradient weight $\sigma (-h)$ goes to zero and the data is spent. Third, overfitting: with one comparison per prompt, empirical preferences are mostly 0 or 1, which Azar et al. (2023) show undermines DPO's KL regularization (the motivation for IPO).

Tajwar et al. (2024) studied this systematically and found that methods using on-policy sampling, and that actively push down the likelihood of some responses (a "negative gradient"), outperform offline and maximum-likelihood objectives, especially when the best responses are far from what the starting model already produces. Online DPO combines both.

Where it came from#

Several groups converged on the idea within a few months.

• Iterative DPO with theory. Xiong, Dong, Ye, Wang, Zhong, Ji, Jiang and Zhang (December 2023, ICML 2024) analyze the KL-regularized RLHF problem as a contextual bandit and show offline methods lack strategic exploration. They propose an iterative DPO algorithm for the online setting with finite-sample guarantees.
• Online AI Feedback (OAIF). Guo et al. (Google DeepMind, February 2024) sample two responses from the current model at every step, ask an LLM annotator which is better, and apply DPO, IPO or SLiC immediately.
• Self-Rewarding Language Models. Yuan et al. (Meta and NYU, January 2024) let the model judge its own candidates with an LLM-as-a-judge prompt, then run DPO on the resulting pairs, for three iterations.
• Iterative Reasoning Preference Optimization. Pang et al. (2024) apply rounds of DPO plus NLL on chain-of-thought samples labeled by answer correctness.
• RLHF Workflow. Dong et al. (2024, TMLR) publish a full open recipe for online iterative RLHF with DPO as the optimizer, including reward models and code.
• Online IPO and Nash optimization. Calandriello et al. (2024) show online IPO is equivalent to Nash Mirror Descent; Rosset et al. (2024) propose Direct Nash Optimization, a batched on-policy algorithm with monotonic improvement guarantees for general preferences.

What unites them is the loop, not the loss: each one replaces a fixed preference dataset with pairs drawn from the model being trained, labeled by something outside the gradient. They differ in how often the loop turns, who labels, how pairs are selected and what the reference is, which is where most of the practical decisions sit.

The objective#

The per-pair loss is unchanged. What changes is the distribution the pairs are drawn from. Offline DPO minimizes

with $\mu$ a fixed behavior distribution. Online DPO draws two samples from the current policy and lets an annotator $A$ order them:

${\theta }^{-}$ denotes the parameters at sampling time, treated as a constant: gradients do not flow through the sampling step or the annotator. OAIF makes this explicit: in online direct alignment the parameters appear in both sampling and the loss, whereas in offline DPO they appear only in the loss and in RLHF only in sampling. In iterative DPO, ${\theta }^{-}$ is held fixed for a whole round:

with $R$ a reward model or judge, $k$ samples per prompt, and optionally all earlier rounds kept in the training set.

Why on-policy pairs help

The DPO gradient, $-\beta \sigma (-h)(\nabla \log {\pi }_{\theta }(y_w)-\nabla \log {\pi }_{\theta }(y_l))$, moves probability between the two responses in the pair. When both come from ${\pi }_{\theta }$, they are by construction where the policy's mass is, so the update changes what the model actually generates. When they come from another model, the update can be satisfied by shifting mass among responses the policy would never produce, while its real outputs stay unexamined. Fresh pairs also arrive with $h\approx 0$ (both are typical samples of the current model), so every new pair carries close to the maximum gradient weight. And the reference model is the same ${\pi }_{\text{ref}}$ throughout, so the KL-regularized objective DPO was derived from is preserved; what improves is that its expectations are now estimated under the right distribution.

What the evidence shows#

OAIF. With PaLM 2-XS as the policy and PaLM 2-L as the annotator, human raters preferred online over offline versions of DPO, IPO and SLiC about 66% of the time on average. On TL;DR, offline DPO's win rate against the SFT model dropped sharply around step 3,500 as it overfit, while online DPO kept improving and passed it after 4,000 steps; in a four-way human comparison against the SFT baseline, RLHF and RLAIF, online DPO was preferred 58% of the time. Because the annotator is a prompted LLM, it is steerable: instructing it to prefer shorter answers cut average response length from about 120 to about 40 while still improving quality over SFT. Settings were $\beta =0.1$ for DPO, 1.0 for IPO and 0.002 for SLiC, sampling temperature 0.9, batch 128, learning rate 5e-7.

Self-Rewarding LMs. Starting from Llama 2 70B, three iterations of self-judged DPO raised AlpacaEval 2.0 win rate against GPT-4 Turbo from 9.94% to 15.38% to 20.44%. The authors also report average generation length growing from 1,092 to 2,552 over the same iterations and flag it as a possible factor: an LLM judge with a length preference will train that preference into the policy.

Iterative RPO. On GSM8K with Llama-2-70B-Chat, four iterations raised accuracy from 55.6% (zero-shot) to 81.6% (88.7% with majority voting over 32 samples); ARC-Challenge went from 77.8% to 86.7%. Offline DPO on the first round's pairs reached only 61.8%, below SFT on gold solutions (63.5%). Each round sampled 30 solutions per problem, paired correct with incorrect answers (about 55k to 60k pairs per round), used the previous iteration's model as reference, and added an NLL term on the winning solution (weight 1), which the authors found crucial.

Production recipes. Llama 3's post-training ran several rounds, each training DPO mostly on preference data collected with the best models from the previous round, so the data "conforms better to the distribution of the policy model". Tulu 3 found that adding on-policy pairs sampled from its own SFT model improved DPO results. RLHF Workflow's Llama-3-8B model, trained with three iterations, reported strong results on AlpacaEval 2, Arena-Hard and MT-Bench from open data alone.

The algorithm, step by step#

A typical iterative recipe, following RLHF Workflow and Iterative RPO:

Input: SFT model pi_0, prompt pool X, annotator R (reward model / judge / verifier),
       rounds T, samples per prompt k, beta
pi_ref = pi_0 (frozen);  pi = pi_0;  history = []
for t in 1..T:
    X_t = next slice of prompts (fresh prompts each round if possible)
    for x in X_t:
        ys = [pi.sample(x, temperature=0.7 to 1.0) for _ in range(k)]
        scores = [R(x, y) for y in ys]
        y_w, y_l = ys[argmax(scores)], ys[argmin(scores)]
        if scores_tie(y_w, y_l): continue           # identical or equal-score pairs carry no signal
        history.append((x, y_w, y_l))
    pi = train_dpo(init=pi, ref=pi_ref, data=history or this round, epochs=1 to 2)
    evaluate pi on held-out prompts with an evaluator different from R

Design choices that vary between recipes:

RLHF Workflow's concrete settings are a reasonable template: 60K prompts split into three iterations of 20K, multiple samples per prompt with best-of-8 versus worst-of-8 pairing, DPO for two epochs per iteration at a peak learning rate of 5e-7 with batch 128 and KL coefficient 0.1, each iteration initialized from the last but always referencing the SFT model. They also observed significant length bias in their reward models and found a length penalty in data selection reduced output length.

A worked example: one round, by hand#

One prompt, $k=4$ samples from the current policy, scored by a reward model: $R=\{0.2,1.4,-0.3,0.9\}$. Best is sample 2, worst is sample 3. Their summed log-probabilities under the current policy are $-61$ and $-58$; under the SFT reference, $-63$ and $-57$. With $\beta =0.1$:

• Log-ratios: chosen $-61-(-63)=2$, rejected $-58-(-57)=-1$. Margin $h=0.1\times (2-(-1))=0.3$.
• Loss $-\log \sigma (0.3)=0.554$; gradient weight $\sigma (-0.3)=0.426$.
• Compare an offline pair the policy has already separated, say $h=4$: weight $\sigma (-4)=0.018$. The fresh on-policy pair pushes 24 times harder.

Notice that the margin here is not zero even though both samples came from the current policy: the policy has moved away from the reference in earlier rounds, and the margin is measured relative to the reference. If the reference were reset to the current model each round (as Iterative RPO does), both log-ratios would start at 0 and so would $h$.

The reward gap also matters for data quality. A best-minus-worst gap of $1.4-(-0.3)=1.7$ is large relative to reward-model noise; if all four samples score within 0.1 of each other, the "preference" is mostly annotator noise. Dropping pairs below a minimum reward gap is a common and cheap filter.

A toy comparison: offline, iterative and online#

We built a task where offline data provably cannot reveal everything. A policy over 500 candidate responses has 8 shared features plus a per-response bias, so it can represent any distribution. True quality is the sum of a part the features capture and a hidden per-response part (features correlate 0.67 with quality). The reference model is concentrated, so many good responses are rare under it. The annotator is a Bradley-Terry judge on true quality. Every method uses DPO with $\beta =0.1$, the same frozen reference, Adam (learning rate 0.02), batch 64 and 1,200 steps, three seeds; reference quality is −0.02 and the best response scores 3.03.

Offline DPO plateaued at 1.74 by step 600 and never moved again: it had fit its 2,000 pairs and had no information about the regions its own policy was moving into. With the same label budget split into rounds sampled from the current policy, quality rose in steps, one step per new round, to about 2.9. Fully online training got there fastest, by step 300, but used about 38 times as many labels.

The "labels used" column shows a practical effect worth planning for. In later rounds the policy had concentrated on a few responses, and even after 200 resampling attempts the iterative runs could not find 500 or 250 pairs with two different responses; the 4- and 8-round runs ended up with 1,666 and 1,767 labels instead of 2,000. In a language model the equivalent is samples that are near-duplicates. Raise the sampling temperature, take more samples per prompt, use fresh prompts each round, or pair against responses from another source when the policy's own samples stop disagreeing. As in our other toys, the higher KL of the on-policy runs bought quality here because the annotator is exact; with a learned reward model, extra KL is also extra room for reward hacking.

Hyperparameters#

Compute and memory#

Training memory per step is DPO's: a trainable policy (about 16 bytes per parameter with Adam in mixed precision), a frozen reference (2 bytes per parameter, or free with LoRA), and activations for two sequences per pair. What online methods add is generation and annotation. Per training pair, you generate $k$ responses (decoding is memory-bandwidth bound and typically dominates wall-clock unless you use a fast inference engine), score $k$ responses with the annotator, and then do a DPO step. A rough per-pair cost model:

where $\ell$ is response length. With $k=8$ and a judge as large as the policy, generation plus judging easily cost several times the DPO step itself. Iterative rounds amortize this well: generate and score in a batch job with an inference engine, then train. Fully online training needs generation inside the training loop, which is why online trainers (TRL's OnlineDPOTrainer, for example) integrate fast inference servers. Compared with PPO, online DPO has no value network and no advantage estimation, so it is simpler and lighter per step, while getting much of the on-policy benefit.

Failure modes#

Reward hacking. On-policy training optimizes against the annotator, so it finds the annotator's blind spots: length, formatting, flattery, confident tone. Self-Rewarding saw length more than double over three iterations. Always evaluate with a held-out judge or human raters, track length, and consider a length penalty in pair selection as RLHF Workflow did.

Pair collapse. As the policy sharpens, samples for a prompt become near-identical and pairs stop carrying signal (visible in our toy). Watch the fraction of dropped ties and the mean reward gap per round.

Self-reinforcing judges. When the policy judges itself, a judge bias gets amplified rather than corrected. Use an external annotator for at least some pairs, or for evaluation.

Drift across rounds. Continuing from the last round's weights while keeping the original reference lets KL accumulate across rounds. Track KL to ${\pi }_0$ and capability benchmarks after every round, not just the target metric. Resetting the reference to the latest model each round (as Iterative RPO does) removes the accumulated anchor; that is a deliberate trade-off, not a free improvement.

Too many epochs per round. Training each round's pairs to convergence reintroduces the offline overfitting the method was meant to avoid. One or two epochs per round is the norm.

Per-round dashboard: reward-model score of fresh samples, held-out judge win rate, mean output length, KL to the SFT model, fraction of ties dropped, mean reward gap of selected pairs, and general capability benchmarks.

When to use it#

In POFE, iterative DPO is set up as rounds: generate from the current checkpoint, score with the project's reward model or judge, filter ties and small gaps, train, and evaluate with a different judge before the next round starts.

Minimal implementation#

An iterative loop needs only a sampler, a scorer and a DPO trainer. A compact sketch with PyTorch-style pieces:

import torch

def build_round(policy, tokenizer, prompts, score_fn, k=8, temperature=0.8, min_gap=0.5):
    pairs = []
    for x in prompts:
        enc = tokenizer(x, return_tensors="pt").to(policy.device)
        with torch.no_grad():
            outs = policy.generate(**enc, do_sample=True, temperature=temperature,
                                   num_return_sequences=k, max_new_tokens=512)
        ys = tokenizer.batch_decode(outs[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)
        scores = torch.tensor([score_fn(x, y) for y in ys])
        hi, lo = int(scores.argmax()), int(scores.argmin())
        if scores[hi] - scores[lo] >= min_gap and ys[hi] != ys[lo]:
            pairs.append({"prompt": x, "chosen": ys[hi], "rejected": ys[lo]})
    return pairs

# for t in range(rounds):
#     pairs = build_round(policy, tok, prompt_slices[t], reward_model_score)
#     policy = run_dpo(policy, ref_model=sft_model, pairs=pairs, beta=0.1, lr=5e-7, epochs=1)

For fully online training, TRL provides an online DPO trainer (under trl.experimental in recent versions) that samples from the policy each step and scores completions with one or more reward functions or reward models:

from trl.experimental.online_dpo import OnlineDPOConfig, OnlineDPOTrainer

args = OnlineDPOConfig(output_dir="online-dpo", beta=0.1, learning_rate=5e-7,
                       temperature=0.9, max_new_tokens=512, bf16=True)
trainer = OnlineDPOTrainer(model=sft_model, reward_funcs=reward_model, args=args,
                           train_dataset=prompts_only, processing_class=tokenizer)
trainer.train()

Further reading#

Each of these papers makes a different design choice in the loop above; reading OAIF and RLHF Workflow together gives the online and iterative ends of the spectrum.

• Guo et al. (2024), Direct Language Model Alignment from Online AI Feedback. Online DPO, IPO and SLiC with an LLM annotator.
• Xiong et al. (2023), Iterative Preference Learning from Human Feedback. Theory for iterative DPO under a KL constraint.
• Dong et al. (2024), RLHF Workflow: From Reward Modeling to Online RLHF. An open, reproducible iterative recipe.
• Yuan et al. (2024), Self-Rewarding Language Models. The model as its own judge.
• Pang et al. (2024), Iterative Reasoning Preference Optimization. Iterative DPO plus NLL for reasoning.
• Tajwar et al. (2024), Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. Why on-policy sampling and negative gradients matter.
• Calandriello et al. (2024), Online Preference Optimisation, and Rosset et al. (2024), Direct Nash Optimization. Online IPO and Nash-style generalizations.
• Xu et al. (2024), Is DPO Superior to PPO for LLM Alignment? The out-of-distribution argument against offline DPO.

Frequently asked questions#

References#

1. Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., Blondel, M. (2024). Direct Language Model Alignment from Online AI Feedback. arxiv.org/abs/2402.04792
2. Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., Zhang, T. (2023). Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint. arxiv.org/abs/2312.11456
3. Dong, H., Xiong, W., Pang, B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., Zhang, T. (2024). RLHF Workflow: From Reward Modeling to Online RLHF. arxiv.org/abs/2405.07863
4. Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., Weston, J. (2024). Self-Rewarding Language Models. arxiv.org/abs/2401.10020
5. Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., Weston, J. (2024). Iterative Reasoning Preference Optimization. arxiv.org/abs/2404.19733
6. Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schneider, J., Xie, T., Ermon, S., Finn, C., Kumar, A. (2024). Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Data. arxiv.org/abs/2404.14367
7. Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Avila Pires, B., Richemond, P. H., Le Lan, C., Valko, M., Liu, T., Joshi, R., Zheng, Z., Piot, B. (2024). Human Alignment of Large Language Models through Online Preference Optimisation. arxiv.org/abs/2403.08635
8. Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadallah, A., Xie, T. (2024). Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences. arxiv.org/abs/2404.03715
9. Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., Wu, Y. (2024). Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study. arxiv.org/abs/2404.10719
10. Gheshlaghi Azar, M., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., Munos, R. (2023). A General Theoretical Paradigm to Understand Learning from Human Preferences. arxiv.org/abs/2310.12036
11. Llama Team, AI @ Meta (Grattafiori, A. et al.) (2024). The Llama 3 Herd of Models. arxiv.org/abs/2407.21783
12. Lambert, N. et al. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. arxiv.org/abs/2411.15124
13. von Werra, L. et al. (Hugging Face) (2020). TRL: Transformer Reinforcement Learning (OnlineDPOTrainer). github.com/huggingface/trl

Related reading