dai2023-linear-attention-equals-gradient-descent-on-linear-layer

IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s3-understanding-in-context-learning.md

Created 2026-08-24T17:10:53+00:00

A linear layer updated by one step of gradient descent, F(x) = W₀x + Σᵢ(eᵢ⊗x′ᵢ)x, is mathematically equivalent to a linear attention operation where error signals are values, historic inputs are keys, and the current input is the query.

Summary

Doing a single gradient-descent update on a simple linear model produces the exact same output as a linear attention mechanism, where past inputs serve as keys, error signals serve as values, and the current input acts as the query. In other words, what looks like a sophisticated attention architecture is really just basic optimization viewed from a different angle, which means tools and intuitions from either field can be carried directly into the other.