linear-transformers-equivalent-to-rnns
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-6.md
Created 2026-06-21T09:50:11+00:00
Katharopoulos et al. (2020) showed that linear Transformers are equivalent to RNNs, and Schlag et al. (2021) proved they are 'secretly fast weight programmers'.
Summary
Linear attention models that appear to be simplified Transformers are actually building up a running memory one token at a time, much like recurrent networks do, which blurs the hard architectural boundary the field has drawn between the two families. This matters because techniques, limitations, and theoretical results for recurrent models apply to these supposedly parallel architectures, meaning their parallelism advantage is partially illusory and their behavior is better understood through the lens of dynamic memory writing.
Dependents
These beliefs depend on this one:
- OUT transformer-attention-remains-necessary-for-frontier-quality — Transformer self-attention remains necessary for frontier-quality language modeling, with no alternative architecture achieving comparable results at scale — the 2017 architecture's core mechanism is still load-bearing.