rlhf-kl-divergence-prevents-reward-hacking

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback.md

Created 2026-06-21T09:50:10+00:00

A KL divergence penalty between the RL-trained policy and the SFT baseline policy is used in RLHF to prevent reward hacking (the model exploiting reward model weaknesses) and to stabilize training.

Summary

During preference training, keeping the model's outputs anchored close to its original fine-tuned version acts as a guardrail against the model finding sneaky shortcuts to score well with the human-feedback evaluator. Without this constraint, the model can exploit gaps in how the evaluator judges responses and the training loop itself becomes unstable, so this penalty is what keeps the system honest and the results reliable.

Dependents

These beliefs depend on this one:

IN rlhf-has-complete-mathematical-pipeline — RLHF has a fully specified mathematical pipeline: Bradley-Terry loss trains the reward model from comparisons, KL divergence constrains policy drift, and PPO's clipped surrogate optimizes the policy — each component addressing a distinct failure mode.