rlhf-kl-penalty-dual-purpose

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-06-21T09:50:10+00:00

The KL divergence penalty in RLHF serves two purposes: (1) prevents the policy from drifting too far from the SFT model (preserving generalization), and (2) encourages high-entropy outputs (preventing mode collapse to canned responses)

Summary

The KL penalty in RLHF acts as a two-part guardrail: it stops the model from overfitting to the reward signal at the expense of its broader language abilities, and it pushes the model to keep generating varied responses instead of collapsing into a small set of safe, repetitive answers. Without it, RLHF training would produce a model that performs well on the specific reward but is brittle, narrow, and boring in practice.

Dependents

These beliefs depend on this one:

IN ppo-employs-triple-independent-stability-mechanism — PPO employs three independent stability mechanisms — probability ratio clipping ([1-ε, 1+ε]), dual-purpose KL divergence penalty (preventing reward hacking and preserving pretraining knowledge), and pretraining gradient mixing (PPO-ptx preventing catastrophic forgetting) — each addressing a distinct failure mode through engineering rather than theoretical resolution.