all-training-objectives-are-compression-variants
IN derived (depth 2)
Created 2026-06-21T13:01:36+00:00 · Reviewed 2026-06-21T14:41:08+00:00
The three main Transformer training task types — masked language modeling (BERT-style bidirectional), autoregressive/causal (GPT-style left-to-right), and PrefixLM (hybrid bidirectional prefix with causal generation) — are all variants of self-supervised compression, and Chinchilla's information-theoretic grounding (model quality directly measures compression capability) reveals they differ in compression strategy rather than in kind.
Justifications
SL — All three training objectives are compression variants unified by information theory
Antecedents (all must be IN):
- IN three-training-task-types-masked-autoregressive-prefixlm — The three main Transformer training task types are masked language modeling (BERT-style), autoregressive/causal language modeling (GPT-style), and prefixLM (T5-style, where a prefix provides context and the model autoregressively predicts the target).
- IN chinchilla-grounds-scaling-in-information-theory — Chinchilla research established that language model quality directly measures data compression capability (compressing ImageNet to 43% vs PNG's 58%), grounding compute-optimal scaling laws in information-theoretic foundations rather than purely empirical curve-fitting.
Dependents
These beliefs depend on this one:
- IN compression-is-the-fundamental-scaling-variable — Since all transformer training objectives — masked, autoregressive, and prefix language modeling — are variants of data compression, and empirical evidence consistently shows data volume outweighs parameter count for model performance, the fundamental scaling variable is the quality and quantity of information being compressed, not the capacity of the compressor.
- IN scaling-laws-generalize-across-training-paradigms — Chinchilla's information-theoretic grounding of scaling in compression quality, combined with the observation that all three training task types (masked, autoregressive, prefix) are compression variants, implies that scaling law relationships should generalize across training paradigms — not just the autoregressive models where they were empirically measured.