pretraining-and-alignment-have-inverted-scaling-priorities
IN derived (depth 2)
Created 2026-06-21T10:00:59+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Pretraining and alignment exhibit inverted scaling priorities: in pretraining, data volume outweighs parameter count (Chinchilla, Llama 1 13B beating GPT-3 175B), while in RLHF reward modeling, scaling model size is more effective than scaling comparison dataset size — suggesting different phases of the training pipeline are bottlenecked by fundamentally different resources.
Summary
The most expensive bottleneck shifts depending on which training phase you are in: during pretraining, feeding a smaller model more text outperforms building a bigger model, but during alignment with human feedback, enlarging the reward model pays off more than collecting more comparison examples. This means a team allocating compute and data budgets needs to treat these two stages as fundamentally different problems rather than applying one uniform "scale everything up" strategy.
Justifications
SL — the dominant scaling lever inverts between pretraining (data) and reward modeling (params)
Antecedents (all must be IN):
- IN data-scaling-outweighs-parameter-scaling — Empirical results consistently show data volume matters more than parameter count: Chinchilla demonstrated models were undertrained, Llama 1 13B beat GPT-3 175B, and Llama 3 8B continued improving at 75x Chinchilla-optimal data.
- IN rlhf-scaling-reward-model-more-effective-than-data — In RLHF, scaling reward model size is more effective for improving performance than scaling the comparison dataset size.
Dependents
These beliefs depend on this one:
- IN capacity-bottleneck-inverts-between-pretraining-and-alignment — The transformer's known over-parameterization at the pretraining level (compression to 60% of parameters with 95% performance retained) combined with alignment's inverse priority (scaling reward model size matters more than data volume) reveals a systematic inversion: data volume is the bottleneck for pretraining while model capacity is the bottleneck for alignment.
- IN sft-one-epoch-confirms-inverted-scaling-dynamics — The InstructGPT paper's deliberate restriction of SFT training to a single epoch — explicitly to avoid overfitting — independently confirms the inverted scaling dynamics between pretraining and alignment: pretraining thrives on data volume (15T+ tokens in Llama 3) while alignment degrades from the same abundance, requiring intentional information restriction at the pipeline transition point.
- IN training-pipeline-maturity-masks-scaling-asymmetry — The LLM training pipeline's maturation into a standardized engineering discipline (SFT → reward model → PPO, with known costs and reproducible stages) masks a fundamental asymmetry: pretraining benefits most from scaling data volume while alignment benefits most from scaling model size — meaning the same pipeline optimizes in opposite directions at different stages, and uniform scaling strategies are suboptimal.