ppo-value-estimator-initialized-from-reward-model

IN premise — summaries/2026/08/24/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-08-24T17:11:23+00:00

In PPO-RLHF, the value estimator V_ξ is initialized from the trained reward model, trained concurrently via squared TD-error loss, and used only during training (not at inference)

Summary

In the PPO-RLHF setup, the value function that helps guide learning borrows its starting point from the reward model's weights and is trained alongside the policy, but it is thrown away the moment training ends. This means the reward model shapes the final output only indirectly, through the weights it leaves behind in the policy, rather than scoring or adjusting text at inference time.