performers-favor-random-feature-attention
IN premise — summaries/2026/08/24/wiki-Transformer_deep_learning_architecture-chunk-6-chunk-1.md
Created 2026-08-24T17:11:26+00:00
Performers (Choromanski et al., 2022, arXiv:2009.14794) use a random-feature (FAVOR+) approximation of softmax attention to reduce the quadratic complexity of standard self-attention.
Summary
The Performers approach shows that you can approximate the expensive softmax attention computation with a randomized shortcut (FAVOR+) that scales roughly linearly with sequence length instead of quadratically. This matters because it makes it feasible to run attention over very long inputs—full books, hours of video, or large codebases—without the memory and compute costs exploding as the sequence grows.