all-training-objectives-are-compression-variants-v2

IN premise

Created 2026-08-24T19:06:57+00:00

The three main Transformer training task types — masked language modeling (BERT-style bidirectional), autoregressive/causal (GPT-style left-to-right), and PrefixLM (hybrid bidirectional prefix with causal generation) — are all forms of self-supervised prediction, and Chinchilla's establishment that model quality directly measures data compression capability provides an information-theoretic lens through which they can be viewed as differing in predictive approach rather than in fundamental kind.

Summary

The three dominant ways of training Transformer models (fill-in-the-blank, left-to-right prediction, and hybrid) are not three fundamentally different tasks but three different strategies for the same underlying job of compressing and predicting text. This matters because it means architectural differences like BERT versus GPT should be evaluated as variations on a single compression objective rather than as incommensurable paradigms, which changes how you compare, combine, or reason about them.