dual-form-derivation-requires-relaxed-linear-attention-no-softmax

IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s2-background.md

Created 2026-08-24T17:10:53+00:00

The dual-form derivation between attention and gradient descent holds under relaxed linear attention (no softmax, no √d scaling); standard Transformer attention is the softmax-weighted generalization of this linear form.

Summary

There is a clean mathematical link between attention and gradient descent, but it is easiest to see and prove in a simplified version where you just take a dot product without any normalization or scaling. Standard Transformer attention is a more elaborate version of that basic linear operation, so the deeper connection to optimization lives in the simpler case and gets wrapped up by the softmax layer on top.