scaling-laws-generalize-across-training-paradigms

IN derived (depth 3)

Created 2026-06-21T13:06:41+00:00 · Reviewed 2026-06-21T14:41:08+00:00

Chinchilla's information-theoretic grounding of scaling in compression quality, combined with the observation that all three training task types (masked, autoregressive, prefix) are compression variants, implies that scaling law relationships should generalize across training paradigms — not just the autoregressive models where they were empirically measured.

Justifications

SL — If scaling laws arise from compression dynamics and all training objectives are compression variants, the laws' functional form should be paradigm-independent

Antecedents (all must be IN):

IN chinchilla-grounds-scaling-in-information-theory — Chinchilla research established that language model quality directly measures data compression capability (compressing ImageNet to 43% vs PNG's 58%), grounding compute-optimal scaling laws in information-theoretic foundations rather than purely empirical curve-fitting.
IN all-training-objectives-are-compression-variants — The three main Transformer training task types — masked language modeling (BERT-style bidirectional), autoregressive/causal (GPT-style left-to-right), and PrefixLM (hybrid bidirectional prefix with causal generation) — are all variants of self-supervised compression, and Chinchilla's information-theoretic grounding (model quality directly measures compression capability) reveals they differ in compression strategy rather than in kind.