rlhf-implementation-diversified-beyond-standard-ppo

IN derived (depth 1)

Created 2026-06-21T10:12:39+00:00 · Reviewed 2026-06-21T14:41:08+00:00

RLHF implementation has diversified beyond the canonical PPO-only approach: Llama 2 introduced rejection sampling before PPO, PPO-ptx adds a pretraining loss to prevent catastrophic forgetting, and the InstructGPT finding that SFT should train for only 1 epoch reveals stage-specific overfitting risks — indicating the "standard" three-stage pipeline actually requires substantial per-stage customization to work.

Summary

The "standard" RLHF pipeline is really a set of loose principles, not a fixed recipe — each stage has its own distinct failure modes, like overfitting in supervised fine-tuning or catastrophic forgetting during preference optimization, that demand different practical fixes. Any team or system treating PPO as the only viable alignment step, or SFT as just "train until it converges," is working from an outdated and incomplete picture of what actually produces a good model.

Justifications

SL — Three independent implementation innovations show the standard RLHF pipeline requires substantial customization

Antecedents (all must be IN):

  • IN llama2-rlhf-rejection-sampling-then-ppo — Llama 2 used rejection sampling followed by PPO for RLHF alignment, departing from PPO-only approaches, with separate reward models for safety and helpfulness
  • IN ppo-ptx-pretraining-gradients — PPO-ptx adds a pretraining language modeling loss (weighted by γ) to the PPO objective to prevent catastrophic forgetting, first introduced in the InstructGPT paper — 'ptx' stands for 'Mixing Pretraining Gradients'
  • IN rlhf-sft-one-epoch-instructgpt — The InstructGPT paper recommends training the SFT stage for only 1 epoch to avoid overfitting.

Dependents

These beliefs depend on this one: