transformer-introduced-2017-attention-is-all-you-need
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-1.md
Created 2026-06-21T09:50:11+00:00
The transformer architecture was introduced in the 2017 paper 'Attention Is All You Need' by Google researchers, with the original model having approximately 100M parameters.
Summary
The Transformer traces back to a single 2017 paper from a small Google team, and the original model was relatively compact at around 100 million parameters. This matters because it fixes a concrete origin point: any later claim about the architecture's intent, its scaling trajectory, or its design lineage has to be measured against this specific, bounded starting design rather than an undifferentiated "deep learning" history.
Dependents
These beliefs depend on this one:
- IN attention-evolved-from-augmentation-to-architecture — Attention evolved from an add-on mechanism augmenting RNNs (Bahdanau 2014) through standalone viability without recurrence (2016) to becoming the entire architecture (Transformer 2017).
- IN transformer-superseded-lstm-via-parallelism — The Transformer ended LSTM's 20-year dominance (1997–2017) by replacing sequential recurrence with parallel self-attention, enabling massive scaling.