llama-training-data-sizes-progression

IN premise — entries/2026/06/21/wiki-LLaMA-chunk-1.md

Created 2026-06-21T09:50:09+00:00

Llama training data sizes: 1.4T tokens (Llama 1) → 2T (Llama 2) → 15T (Llama 3) → 40T (Llama 4 Scout)

Summary

Meta's Llama models have been trained on rapidly growing amounts of text, from about 1.4 trillion tokens in the first generation up to roughly 40 trillion tokens by Llama 4 Scout, a roughly 30x jump. This matters because it tells you the system is treating raw data scale as a primary axis of progress across model generations, so any comparison of capability improvements has to account for how much more "reading" each newer model did.

Dependents

These beliefs depend on this one:

IN llama-optimized-architecture-for-data-scaling-strategy — Llama's architecture adopts proven refinements from prior independent research (SwiGLU from Shazeer 2020, RoPE from Su et al. 2021, RMSNorm from Zhang & Sennrich 2019) rather than introducing novel components, while its training data volume scaled dramatically from 1.4T to 40T tokens across four generations — a pattern consistent with concentrating effort on data scaling over architectural novelty.