lsa-aligns-better-with-gd-than-softmax
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s3-trained-transformersdo-mimic-gradient.md
Created 2026-08-24T17:11:05+00:00
Trained Transformers align more closely with the analytical GD construction under linear self-attention than under softmax attention, suggesting softmax introduces a perturbation to the ideal GD dynamics, and the effective learning rate η is encoded in the diagonal entry of P·W_V corresponding to the output dimension.
Summary
When you compare what trained transformers actually do against a clean mathematical model of gradient descent, the match is tighter under linear attention than under the standard softmax attention, which points to softmax as the component that bends learning away from its ideal form. It also means the effective learning rate is not just a dial set from outside the model but is encoded in particular entries of the weight matrices themselves, so changing those entries changes how fast the system learns.