lsa-layer-exact-gd-step-weight-construction
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-sR-references.md
Created 2026-08-24T17:11:05+00:00
A single-head linear self-attention layer implementing one GD step on L(W) = (1/2N)Σ‖Wxᵢ − yᵢ‖² uses the block construction W_K = W_Q = diag(I_x, 0), W_V = [[0,0],[W₀,−I_y]], and P = (η/N)·I, yielding the update eⱼ = (xⱼ, yⱼ) + (0, −ΔW·xⱼ).
Summary
This establishes a concrete recipe showing that a single-head linear attention layer, with specific choices of key, query, and value matrices, can exactly perform one gradient descent step on a linear regression loss. It matters because it turns the attention mechanism from a black-box operation into something the system can reason about in terms of classical optimization: the layer's output is literally the GD update, so downstream analysis can treat it as such.