ppo-clipped-surrogate-ratio

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-06-21T09:50:10+00:00

PPO uses a clipped probability ratio π/π_old constrained to [1−ε, 1+ε] to prevent destructively large policy updates, where ε is a hyperparameter controlling the clipping range

Summary

PPO acts as a speed limit on policy updates by capping how much the probability of an action can shift in a single training step relative to the previous version of the policy. This matters because without that cap, one noisy or oversized gradient could catastrophically overwrite a well-learned policy in an instant, making training unstable and hard to recover from.

Dependents

These beliefs depend on this one:

IN ppo-employs-triple-independent-stability-mechanism — PPO employs three independent stability mechanisms — probability ratio clipping ([1-ε, 1+ε]), dual-purpose KL divergence penalty (preventing reward hacking and preserving pretraining knowledge), and pretraining gradient mixing (PPO-ptx preventing catastrophic forgetting) — each addressing a distinct failure mode through engineering rather than theoretical resolution.
IN rlhf-has-complete-mathematical-pipeline — RLHF has a fully specified mathematical pipeline: Bradley-Terry loss trains the reward model from comparisons, KL divergence constrains policy drift, and PPO's clipped surrogate optimizes the policy — each component addressing a distinct failure mode.