transformer-three-architectural-variants

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-1.md

Created 2026-06-21T09:55:54+00:00

Transformers have three architectural variants: encoder-only (e.g., BERT, for representation), decoder-only (e.g., GPT, for autoregressive generation), and encoder-decoder (e.g., T5, for sequence-to-sequence).

Dependents

These beliefs depend on this one:

• IN bert-gpt-divergence-from-common-ancestor — BERT and GPT represent a speciation event from a common ancestor: both derived from the 2017 Transformer in the same year (2018), but chose opposite pretraining strategies (masked vs. causal LM) and opposite architectural halves (encoder vs. decoder), yielding fundamentally different capability profiles.
• IN transformer-architecture-unified-yet-specialized — The Transformer is simultaneously a unified architecture (all variants built on multi-head self-attention) and a specialized one (encoder-only, decoder-only, encoder-decoder serve distinct task families).