Open the workspace
Glossary

KL divergence penalty

Why post-training penalises KL divergence from a reference model, how to read KL in your training logs, and typical values for DPO and GRPO.

POFE ResearchPublished 2026-09-05Updated 2026-10-304 min read

KL divergence penalty: A regulariser that penalises the trained policy for moving away from a reference model, measured by KL divergence.

Explained#

Almost every preference and RL method optimises reward minus a penalty on divergence from a reference policy, usually the SFT model:

The penalty keeps everything the reward says nothing about — fluency, formatting, general knowledge — close to where it was, and limits how far the policy can go to exploit reward gaps. In DPO it appears implicitly through β; in PPO and GRPO as an explicit coefficient or as a per-token penalty in the reward.

Reading it in your logs

  • KL is measured in nats per sequence (or per token — check which). Per-sequence KL grows with response length.
  • Steadily rising KL with flat eval scores suggests the policy is drifting without improving.
  • A sudden jump usually means a few responses collapsed onto a degenerate pattern; inspect samples.
  • In our DPO simulation, moving β from 1.0 to 0.1 raised final KL from about 0.3 to about 4.8 nats with almost no change in preference accuracy.

References#

  1. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. arXiv:2203.02155. arxiv.org/abs/2203.02155
Direct Preference Optimization (DPO)GlossaryReward hackingGlossaryWhat DPO's β actually controls — and why chosen likelihoods fallResearchWhat is post-training? A complete guide for LLM teamsPillar guide

Bring a model and a metric.

Tell us what the model should get better at. A post-training engineer replies within one business day with a plan, a compute estimate and an eval design.