decoder-only-won-the-scaling-race
IN derived (depth 1)
Created 2026-06-21T09:52:14+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Decoder-only architecture became the dominant paradigm for large language models, with GPT, Llama, and most frontier models choosing autoregressive causal modeling over encoder-only or encoder-decoder variants.
Summary
The industry converged on the "predict the next token one at a time" design (used by GPT, Llama, and most frontier models) over the bidirectional or read-then-generate alternatives, meaning virtually every major language model today shares the same fundamental training logic. This simplifies the ecosystem and tooling, but it also locks in specific tradeoffs around how models use context and what kinds of tasks they handle most naturally.
Justifications
SL — Of three transformer variants, decoder-only prevailed for generative scaling while encoder-only (BERT) remained specialized for understanding tasks
Antecedents (all must be IN):
- IN llama-decoder-only-autoregressive — Llama is a decoder-only autoregressive transformer architecture (like GPT-3), not encoder-decoder or encoder-only
- IN llama-decoder-only-autoregressive — Llama is a decoder-only autoregressive transformer architecture (like GPT-3), not encoder-decoder or encoder-only
- IN transformer-three-architecture-variants — The three main Transformer architecture variants are encoder-only (e.g., BERT, bidirectional), decoder-only (e.g., GPT, autoregressive/causal), and encoder-decoder (e.g., T5, original Transformer).
- IN llama-decoder-only-autoregressive — Llama is a decoder-only autoregressive transformer architecture (like GPT-3), not encoder-decoder or encoder-only
Dependents
These beliefs depend on this one:
- IN bert-paradigm-survived-its-own-architectural-obsolescence — BERT's pretrain-then-fine-tune paradigm persisted even as decoder-only models superseded encoder-only architectures in the scaling race — the methodology that BERT proved was inherited by the very architecture class that replaced it, demonstrating that methodological contributions can outlast the architectures that introduce them.
- IN decoder-only-dominance-driven-by-data-scaling-advantage — Decoder-only architectures became dominant in the scaling era, coinciding with evidence that data volume matters more than parameter count for model performance. Autoregressive next-token prediction's ability to leverage massive unsupervised text corpora may have been a contributing factor in this dominance.
- OUT transformer-attention-remains-necessary-for-frontier-quality — Transformer self-attention remains necessary for frontier-quality language modeling, with no alternative architecture achieving comparable results at scale — the 2017 architecture's core mechanism is still load-bearing.