rlhf-three-stage-pipeline-is-standard-alignment

OUT derived (depth 1)

Created 2026-06-21T09:52:15+00:00

RLHF's three-stage pipeline (SFT → reward model → PPO) remains the standard proven alignment method, having produced both InstructGPT and ChatGPT.

Justifications

SL — RLHF's status as the standard approach is gated on whether direct alignment methods (which eliminate its core components) prove sufficient replacements

Antecedents (all must be IN):

IN rlhf-three-stage-pipeline — The RLHF training pipeline has three stages: (1) supervised fine-tuning of a base model, (2) training a reward model from human preference rankings, and (3) optimizing the policy against the reward model using RL (typically PPO).
IN rlhf-applied-after-gpt3-instructgpt-chatgpt — RLHF was applied after GPT-3 to create InstructGPT, then ChatGPT (launched November 30, 2022)

Unless (any of these IN defeats this justification):

IN dpo-eliminates-reward-model-and-rl — Direct Preference Optimization (DPO) eliminates both the separate reward model and the RL loop, collapsing RLHF into a single supervised learning loss over preference pairs via a change-of-variables reparameterization where the partition function cancels in pairwise comparisons