rlhf-sft-one-epoch-instructgpt

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback.md

Created 2026-06-21T09:50:10+00:00

The InstructGPT paper recommends training the SFT stage for only 1 epoch to avoid overfitting.

Summary

The InstructGPT recipe limits supervised fine-tuning to a single pass through the labeled examples because going through them again makes the model memorize the specific demonstrations instead of learning generalizable behavior. This keeps the SFT stage deliberately short and lightweight, shifting most of the quality signal into the later preference-learning step rather than relying on brute-force repetition of examples.

Dependents

These beliefs depend on this one:

IN rlhf-implementation-diversified-beyond-standard-ppo — RLHF implementation has diversified beyond the canonical PPO-only approach: Llama 2 introduced rejection sampling before PPO, PPO-ptx adds a pretraining loss to prevent catastrophic forgetting, and the InstructGPT finding that SFT should train for only 1 epoch reveals stage-specific overfitting risks — indicating the "standard" three-stage pipeline actually requires substantial per-stage customization to work.
IN sft-one-epoch-confirms-inverted-scaling-dynamics — The InstructGPT paper's deliberate restriction of SFT training to a single epoch — explicitly to avoid overfitting — independently confirms the inverted scaling dynamics between pretraining and alignment: pretraining thrives on data volume (15T+ tokens in Llama 3) while alignment degrades from the same abundance, requiring intentional information restriction at the pipeline transition point.