daa-beta-controls-deviation-from-reference
IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-3.md
Created 2026-06-21T09:50:10+00:00
The β parameter in all Direct Alignment Algorithms (DPO, IPO, KTO) serves as KL regularization strength controlling deviation from the reference SFT policy — higher β keeps the policy closer to the reference
Summary
The beta parameter in alignment methods like DPO, IPO, and KTO acts as a safety dial: it sets how far the model's behavior is allowed to drift from its original, pre-alignment version. Cranking beta higher keeps the model closer to what it already knew, trading alignment aggressiveness for stability, so it's the key knob for balancing how much personality the alignment process is allowed to reshape.
Dependents
These beliefs depend on this one:
- IN direct-alignment-eliminated-rlhf-complexity — DPO, IPO, and KTO independently eliminated RLHF's separate reward model and RL loop while sharing a common β-controlled KL regularization structure, each varying only the loss function and data requirements.