kl-regularization-prevents-reward-model-overfitting
IN premise — summaries/2026/08/24/wiki-Reinforcement_learning_from_human_feedback-chunk-1.md
Created 2026-08-24T17:11:22+00:00
KL regularization in RLHF constrains the learned policy from straying too far from the unaligned base model, reducing overfitting to the reward model and stabilizing training.
Summary
A built-in penalty keeps a fine-tuned model close to its original starting point so it cannot exploit gaps or quirks in the human-feedback scoring system. Without this constraint, the model learns to game the scorer rather than actually become more helpful, and training becomes unstable.