rlhf-training-loop-ascent-descent-split
IN premise — summaries/2026/08/24/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md
Created 2026-08-24T17:11:23+00:00
In the RLHF outer training loop, the PPO policy objective is optimized by gradient ascent (maximizing the clipped surrogate) while the value estimator loss is optimized by gradient descent (minimizing squared TD-error)
Summary
The RLHF training loop drives its two main components in opposite directions: the policy model is pushed to improve (climbing toward higher predicted reward), while the value estimator is pushed to become more accurate (falling toward smaller prediction errors). This matters because the loop is not a single unified "minimize a loss" step; it is a push-pull structure where the two parts work against each other, and getting either direction wrong destabilizes the whole training process.