compression-is-the-fundamental-scaling-variable

IN derived (depth 3)

Created 2026-06-21T13:06:41+00:00 · Reviewed 2026-06-21T14:41:08+00:00

Since all transformer training objectives — masked, autoregressive, and prefix language modeling — are variants of data compression, and empirical evidence consistently shows data volume outweighs parameter count for model performance, the fundamental scaling variable is the quality and quantity of information being compressed, not the capacity of the compressor.

Justifications

SL — If all training is compression and more data beats more parameters, then the information supply — not the model architecture — is the binding constraint

Antecedents (all must be IN):

IN all-training-objectives-are-compression-variants — The three main Transformer training task types — masked language modeling (BERT-style bidirectional), autoregressive/causal (GPT-style left-to-right), and PrefixLM (hybrid bidirectional prefix with causal generation) — are all variants of self-supervised compression, and Chinchilla's information-theoretic grounding (model quality directly measures compression capability) reveals they differ in compression strategy rather than in kind.
IN data-scaling-outweighs-parameter-scaling — Empirical results consistently show data volume matters more than parameter count: Chinchilla demonstrated models were undertrained, Llama 1 13B beat GPT-3 175B, and Llama 3 8B continued improving at 75x Chinchilla-optimal data.

Dependents

These beliefs depend on this one:

IN tokenization-is-critical-unsettled-compression-bottleneck — Since compression is the fundamental scaling variable and the input representation layer (tokenization strategy and positional encoding) remains actively contested, tokenization may be among the most impactful yet least settled components of the scaling pipeline — the compression process begins at the input layer where raw data is first reduced, making this contested design space potentially significant for scaling outcomes.