llama2-rlhf-rejection-sampling-then-ppo

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md

Created 2026-06-21T09:50:09+00:00

Llama 2 used rejection sampling followed by PPO for RLHF alignment, departing from PPO-only approaches, with separate reward models for safety and helpfulness

Summary

Instead of relying solely on reinforcement learning to shape responses, Llama 2 first filters out clearly bad outputs before fine-tuning, and it trains two separate scoring models—one for safety and one for helpfulness—so neither objective gets sacrificed to the other. This matters because the alignment is a layered, defensive process rather than a single optimization pass, meaning safety is structurally protected in the architecture rather than just a competing reward signal that could be overridden.

Dependents

These beliefs depend on this one:

IN rlhf-implementation-diversified-beyond-standard-ppo — RLHF implementation has diversified beyond the canonical PPO-only approach: Llama 2 introduced rejection sampling before PPO, PPO-ptx adds a pretraining loss to prevent catastrophic forgetting, and the InstructGPT finding that SFT should train for only 1 epoch reveals stage-specific overfitting risks — indicating the "standard" three-stage pipeline actually requires substantial per-stage customization to work.