attention-mechanism-predates-transformer
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-6.md
Created 2026-06-21T09:50:10+00:00
The attention mechanism predates the Transformer: Bahdanau et al. (2014) introduced additive neural attention for machine translation, and Luong et al. (2015) refined it with multiplicative attention.
Summary
Attention in neural networks was not invented by the Transformer; it was already being used to help translation models focus on the relevant parts of a sentence before 2017. This matters because it means the Transformer's real breakthrough was turning attention inward onto itself and eliminating the sequential recurrence, not introducing attention as a concept in the first place.
Dependents
These beliefs depend on this one:
- IN attention-evolved-from-augmentation-to-architecture — Attention evolved from an add-on mechanism augmenting RNNs (Bahdanau 2014) through standalone viability without recurrence (2016) to becoming the entire architecture (Transformer 2017).