Direct Preference Optimization (DPO): A preference-learning method that trains directly on chosen/rejected response pairs, optimising the RLHF objective in closed form.
Explained#
Direct Preference Optimization (Rafailov et al., 2023) trains a policy on pairs of responses to the same prompt, one preferred () and one rejected (). It optimises the same KL-regularised objective as RLHF, but by rewriting the reward in terms of the policy itself it removes the separate reward model and the online sampling loop:
β sets how strongly the policy is held to the reference model; 0.1 is a common default. Lower β lets the model move further for the same preferences.
What practitioners trip over
- Held-out preference accuracy is a weak signal: in our simulation it varied by under two points across a 100× range of β while KL and true reward changed dramatically.
- The probability of chosen responses usually falls during training (likelihood displacement); it becomes a problem when mass moves to undesired outputs.
- Near-duplicate pairs carry little signal and increase displacement.
Variants: IPO (regularised against overfitting deterministic preferences), KTO (unpaired thumbs-up/down data), ORPO and SimPO (no reference model). Deep dive: DPO β and likelihood displacement.
References#
- Rafailov, R. et al. (2023). Direct Preference Optimization. arXiv:2305.18290. arxiv.org/abs/2305.18290
- Razin, N. et al. (2024). Unintentional Unalignment: Likelihood Displacement in DPO. arXiv:2410.08847. arxiv.org/abs/2410.08847