performer-gram-schmidt-orthonormalizes-random-features
IN premise — summaries/2026/08/24/wiki-Transformer_deep_learning_architecture-chunk-4-chunk-2.md
Created 2026-08-24T17:11:26+00:00
The Performer refines Random Feature Attention by sampling w₁,…,w_D from N(0, σ²I) and then applying the Gram-Schmidt process to orthonormalize them, yielding a more precise approximation of softmax attention as Q(KᵀV/√dₖ)
Summary
The Performer takes randomly generated feature vectors and forces them to be mutually perpendicular and unit-length, a step that measurably tightens how closely the mechanism approximates true softmax attention. In practical terms, this orthonormalization step is what keeps the linear-time shortcut close enough to the exact computation to be useful, rather than just a coarse random guess.