dai2023-duality-requires-relaxed-linear-attention
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s3-understanding-in-context-learning.md
Created 2026-08-24T17:10:53+00:00
The equivalence between Transformer attention and gradient descent holds specifically in the relaxed linear attention form (no softmax, no √d scaling); standard softmax attention is an approximation to this form.
Summary
The neat mathematical link between attention and gradient descent only holds exactly in a simplified, softmax-free version of attention; the attention actually used in standard Transformers is a rougher approximation of this cleaner form. This matters because it tells us which theoretical guarantees are solid (linear attention) and which are approximate (the softmax-based attention most deployed models rely on), setting the stage for understanding where the theory can and cannot be trusted.