transformer-three-architecture-variants

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture.md

Created 2026-06-21T09:50:10+00:00

The three main Transformer architecture variants are encoder-only (e.g., BERT, bidirectional), decoder-only (e.g., GPT, autoregressive/causal), and encoder-decoder (e.g., T5, original Transformer).

Summary

The landscape of Transformer-based models breaks down into three fundamental building patterns: one that ingests all input at once to understand context, one that produces output word by word in a predictive chain, and one that combines both reading and generation. This taxonomy matters because it determines what a model can and cannot do, and any reasoning about model capabilities, training strategies, or task fit has to start from knowing which of these patterns is in play.

Dependents

These beliefs depend on this one:

IN decoder-only-won-the-scaling-race — Decoder-only architecture became the dominant paradigm for large language models, with GPT, Llama, and most frontier models choosing autoregressive causal modeling over encoder-only or encoder-decoder variants.