transformer-replaced-recurrence-with-self-attention-2017

IN premise — entries/2026/06/21/wiki-Recurrent_neural_network-chunk-6.md

Created 2026-06-21T09:55:53+00:00

The Transformer (Vaswani et al. 2017, 'Attention is All You Need') replaced recurrence with self-attention and became dominant for sequence modeling.

Dependents

These beliefs depend on this one:

• IN attention-bridge-rnn-to-transformer — The attention mechanism served as the transitional innovation from RNN-based to Transformer-based architectures, first augmenting encoder-decoder RNNs to solve the fixed-size bottleneck, then replacing recurrence entirely.
• OUT rnn-theoretical-power-practically-accessible — RNNs' theoretical computational power (Turing-completeness, context-sensitive language recognition) is practically accessible for sequence modeling tasks, given LSTM's ability to learn over thousands of timesteps.
• IN sequence-modeling-three-generations — Sequence modeling evolved through three generations — RNNs, LSTMs, Transformers — each solving a specific limitation of its predecessor (vanishing gradients for LSTMs, sequential bottleneck for Transformers).