momentum-based-attention-outperforms-vanilla-attention

IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s0-abstract.md

Created 2026-08-24T17:10:53+00:00

The momentum-based attention variant, designed by analogy with SGD-with-momentum (Polyak 1964; Sutskever et al. 2013), yields consistent performance gains over vanilla attention on both language modeling and ICL benchmarks.

Summary

Adding a momentum mechanism to how attention weights are computed reliably makes the model perform better than standard attention, on both predicting the next word and learning from examples given in context. This matters because it means the improvement is general rather than tied to a specific task, so the momentum-based variant is the safer default choice when building or comparing attention layers.