Reward hacking: When a policy learns to score highly on its reward function by exploiting gaps in it rather than by solving the intended task.
Explained#
Reward hacking happens because RL optimises exactly the reward you wrote, not the one you meant. The more optimisation pressure — more steps, lower KL penalty — the more likely the policy finds the gap.
Examples we see in post-training
- Verifier gaps: a SQL reward that checks the query runs, not that it returns the right rows; code rewards where tests can be special-cased.
- Judge exploitation: longer, more confident answers that an LLM judge rates higher regardless of correctness (verbosity bias, Zheng et al., 2023).
- Format gaming: a "contains a citation" reward satisfied by citing irrelevant documents.
- Policy gaming: a support agent rewarded for "resolved" that resolves by offering refunds it shouldn't.
Detecting and preventing it
- Track reward and an independent held-out eval together. Reward rising while the eval is flat is the signature.
- Read samples from the top of the reward distribution every few hundred steps.
- Combine rewards: correctness + format + explicit penalties for known exploits.
- Keep a KL penalty to the reference model and cap training length.
- Prefer verifiable rewards; calibrate judges (why κ isn't enough).
References#
- Zheng, L. et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685. arxiv.org/abs/2306.05685