single-attention-layer-exactly-one-gd-step
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-abstract.md
Created 2026-08-24T17:11:04+00:00
A single linear self-attention layer with a specific weight construction is exactly equivalent to one gradient-descent step on a mean-squared-error regression loss, and a stack of k such layers equals k GD steps with iterative curvature correction (von Oswald et al., ICML 2023, arXiv:2212.07677v2).
Summary
Each linear attention layer, with its weights set up in a particular way, performs exactly the same computation as one step of gradient descent on a basic least-squares fit, so a stack of such layers is nothing more than a fixed number of gradient-descent steps with a built-in correction at each one. This matters because it turns the opaque "attention" operation into a concrete optimization trajectory, giving us a precise mathematical lens for predicting what these layers compute, where they succeed, and where they diverge from the optimization problem they mirror.