bert-gpt-divergence-from-common-ancestor

IN derived (depth 1)

Created 2026-06-21T10:06:02+00:00 · Reviewed 2026-06-21T15:37:01+00:00

BERT and GPT represent a speciation event from a common ancestor: both derived from the 2017 Transformer in the same year (2018), but chose opposite pretraining strategies (masked vs. causal LM) and opposite architectural halves (encoder vs. decoder), yielding fundamentally different capability profiles.

Justifications

SL — Same base architecture, same year, opposite design choices — a natural experiment in ML architecture

Antecedents (all must be IN):

IN bert-masked-lm-gpt-causal-lm-same-year-2018 — BERT (2018) uses masked language modeling (bidirectional) while GPT (2018) uses causal language modeling (autoregressive left-to-right); both published the same year with opposite pre-training strategies
IN transformer-three-architectural-variants — Transformers have three architectural variants: encoder-only (e.g., BERT, for representation), decoder-only (e.g., GPT, for autoregressive generation), and encoder-decoder (e.g., T5, for sequence-to-sequence).
IN encoder-only-decoder-only-encoder-decoder-variants — BERT is encoder-only, GPT series is decoder-only, T5 is encoder-decoder — all are Transformer architecture variants

Dependents

These beliefs depend on this one:

IN transformer-flexibility-plus-hardware-enabled-rapid-speciation — Transformer dominance stems from a unique combination of architectural flexibility and hardware synergy — the unified attention mechanism enabled rapid speciation into encoder-only (BERT) and decoder-only (GPT) variants within one year of the original paper, while GPU-friendly parallelism eliminated the compute bottleneck that constrained all RNN-based predecessors.