ppo-ptx-pretraining-gradients

IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback-chunk-2.md

Created 2026-06-21T09:50:10+00:00

PPO-ptx adds a pretraining language modeling loss (weighted by γ) to the PPO objective to prevent catastrophic forgetting, first introduced in the InstructGPT paper — 'ptx' stands for 'Mixing Pretraining Gradients'

Summary

When fine-tuning a language model with reinforcement learning, the model risks losing the general language knowledge it spent billions of tokens learning. PPO-ptx guards against that by mixing a small dose of the original next-word-prediction training signal into the reinforcement learning objective, so the model keeps its base language skills while it learns to follow instructions better.

Dependents

These beliefs depend on this one:

IN ppo-employs-triple-independent-stability-mechanism — PPO employs three independent stability mechanisms — probability ratio clipping ([1-ε, 1+ε]), dual-purpose KL divergence penalty (preventing reward hacking and preserving pretraining knowledge), and pretraining gradient mixing (PPO-ptx preventing catastrophic forgetting) — each addressing a distinct failure mode through engineering rather than theoretical resolution.
IN rlhf-implementation-diversified-beyond-standard-ppo — RLHF implementation has diversified beyond the canonical PPO-only approach: Llama 2 introduced rejection sampling before PPO, PPO-ptx adds a pretraining loss to prevent catastrophic forgetting, and the InstructGPT finding that SFT should train for only 1 epoch reveals stage-specific overfitting risks — indicating the "standard" three-stage pipeline actually requires substantial per-stage customization to work.