llama3-8b-trained-15t-tokens-chinchilla-suboptimal

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md

Created 2026-06-21T09:50:09+00:00

Llama 3 8B was trained on 15T tokens — 75x more than the Chinchilla-optimal 200B tokens — and performance continued to scale log-linearly, challenging Chinchilla scaling assumptions

Summary

Meta trained an 8-billion-parameter model on roughly 75 times more data than the standard Chinchilla guideline would call "optimal," yet the model kept improving instead of plateauing. This matters because it suggests the usual scaling rules of thumb may be too conservative — teams can spend a lot of extra compute overtraining a smaller model and still see real gains, which shifts the cost calculus when deciding between building a bigger model or just training a smaller one longer.

Dependents

These beliefs depend on this one:

OUT chinchilla-balance-prescribes-optimal-resource-allocation — Chinchilla's prescription to scale parameters and data in equal proportion provides the optimal training resource allocation strategy.
OUT chinchilla-information-theory-provides-principled-scaling-guidance — Chinchilla's information-theoretic foundation — establishing that language modeling is compression and predicting parameter redundancy — provides principled guidance for model scaling decisions rather than purely empirical groping, offering a theoretical compass for the craft discipline's otherwise empirically-driven methodology.
IN data-scaling-outweighs-parameter-scaling — Empirical results consistently show data volume matters more than parameter count: Chinchilla demonstrated models were undertrained, Llama 1 13B beat GPT-3 175B, and Llama 3 8B continued improving at 75x Chinchilla-optimal data.
IN scaling-predictions-fail-at-both-capability-and-resource-levels — Scaling predictions systematically fail at both capability and resource allocation levels: emergent abilities appear discontinuously at unpredictable thresholds rather than following smooth power-law trends, and Chinchilla-optimal compute allocation is systematically violated by successful models (Llama 3 8B trained at 75x the prescribed data-to-parameter ratio with continued improvement) — the field's quantitative scaling framework provides trend guidance but not operational prediction.