rlhf-three-stage-pipeline

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback.md

Created 2026-06-21T09:50:10+00:00

The RLHF training pipeline has three stages: (1) supervised fine-tuning of a base model, (2) training a reward model from human preference rankings, and (3) optimizing the policy against the reward model using RL (typically PPO).

Summary

Alignment in large language models works as a sequential three-step process: teach the model to produce well-formed responses, learn what humans prefer from comparison data, then use reinforcement learning to push the model toward those preferences. This staging matters because problems can be isolated to a specific step, and improvements in one stage only help if the others are sound.

Dependents

These beliefs depend on this one:

IN rlhf-has-complete-mathematical-pipeline — RLHF has a fully specified mathematical pipeline: Bradley-Terry loss trains the reward model from comparisons, KL divergence constrains policy drift, and PPO's clipped surrogate optimizes the policy — each component addressing a distinct failure mode.
OUT rlhf-three-stage-pipeline-is-standard-alignment — RLHF's three-stage pipeline (SFT → reward model → PPO) remains the standard proven alignment method, having produced both InstructGPT and ChatGPT.