linear-attention-equivalence-to-fast-weight-rnn

IN premise — summaries/2026/08/24/wiki-Transformer_deep_learning_architecture-chunk-6-chunk-1.md

Created 2026-08-24T17:11:26+00:00

Katharopoulos et al. (ICML 2020) demonstrated that Transformers with linear attention are equivalent to RNNs, and Schlag, Irie & Schmidhuber (ICML 2021) proved linear-attention Transformers compute the same function as fast-weight programmers, with lineage tracing to Schmidhuber (1992) and Hinton & Plaut (1987).

Summary

The math shows that a Transformer using linear attention is doing exactly the same computation as a recurrent network that updates its internal weights as it processes tokens, meaning the attention-versus-recurrence split many people treat as a fundamental divide between architecture families is actually just two descriptions of the same machine. This collapses a major conceptual distinction in deep learning and suggests the recent Transformer revolution largely re-expressed recurrent ideas that trace back to the late 1980s, so the two camps were never as separate as the field assumed.