chinchilla-compute-optimal-training
IN premise — entries/2026/06/21/wiki-Generative_pre-trained_transformer-chunk-2.md
Created 2026-06-21T09:50:09+00:00
Hoffmann et al. (2022, 'Chinchilla', arXiv:2203.15556) showed prior models were undertrained relative to dataset size and that optimal training requires scaling data proportionally with parameters
Summary
The Chinchilla result means that most large models released before 2022 were trained with more compute than their dataset could absorb, so they were effectively overfitting or wasting budget. For this system, it sets a baseline: any training plan that scales parameters without a matching increase in unique data is misallocating resources, and compute spending should be treated as a two-sided constraint (parameters and data) rather than a one-sided push toward bigger models.
Dependents
These beliefs depend on this one:
- OUT chinchilla-balance-prescribes-optimal-resource-allocation — Chinchilla's prescription to scale parameters and data in equal proportion provides the optimal training resource allocation strategy.
- IN chinchilla-grounds-scaling-in-information-theory — Chinchilla research established that language model quality directly measures data compression capability (compressing ImageNet to 43% vs PNG's 58%), grounding compute-optimal scaling laws in information-theoretic foundations rather than purely empirical curve-fitting.