rlhf-scaling-reward-model-more-effective-than-data
IN premise — entries/2026/06/21/wiki-Reinforcement_learning_from_human_feedback.md
Created 2026-06-21T09:50:10+00:00
In RLHF, scaling reward model size is more effective for improving performance than scaling the comparison dataset size.
Summary
When training an AI to follow human preferences, the payoff comes more from making the "judge" model (the reward model) bigger and more capable than from collecting more side-by-side comparison examples. This means the system should channel its compute and engineering effort into a stronger evaluator rather than into ever-larger datasets, since that is where the performance gains actually live.
Dependents
These beliefs depend on this one:
- IN pretraining-and-alignment-have-inverted-scaling-priorities — Pretraining and alignment exhibit inverted scaling priorities: in pretraining, data volume outweighs parameter count (Chinchilla, Llama 1 13B beating GPT-3 175B), while in RLHF reward modeling, scaling model size is more effective than scaling comparison dataset size — suggesting different phases of the training pipeline are bottlenecked by fundamentally different resources.