three-training-task-types-masked-autoregressive-prefixlm
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture.md
Created 2026-06-21T09:50:11+00:00
The three main Transformer training task types are masked language modeling (BERT-style), autoregressive/causal language modeling (GPT-style), and prefixLM (T5-style, where a prefix provides context and the model autoregressively predicts the target).
Summary
This is the foundational taxonomy that says the major ways we train Transformer models come down to just three patterns: fill in a missing word, predict the next word, or generate a target sequence given a setup phrase. It matters to the system because it provides a closed set of categories, so any downstream claim about how a model was trained or how to adapt one should map onto one of these three, and anything claiming a "fourth type" is either genuinely novel or confused.
Dependents
These beliefs depend on this one:
- IN all-training-objectives-are-compression-variants — The three main Transformer training task types — masked language modeling (BERT-style bidirectional), autoregressive/causal (GPT-style left-to-right), and PrefixLM (hybrid bidirectional prefix with causal generation) — are all variants of self-supervised compression, and Chinchilla's information-theoretic grounding (model quality directly measures compression capability) reveals they differ in compression strategy rather than in kind.