two-layer-sa-circuit-copy-then-gd
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s4-do-self-attention-layers-build-regression.md
Created 2026-08-24T17:11:05+00:00
A two-layer self-attention Transformer matches one-step (not two-step) gradient descent performance because the first layer is consumed by a copying operation that merges alternating input tokens e₂ⱼ=(xⱼ) and output tokens e₂ⱼ₊₁=(0,yⱼ) into the concatenated form required for the GD computation in the second layer.
Summary
The first layer of the two-layer Transformer doesn't actually compute anything useful; it just shuffles and recombines input and output tokens into the exact format the second layer expects, which is why the whole stack can only replicate a single step of gradient descent rather than two. This means that if you want to express more steps of optimization with this architecture, you have to spend extra layers just doing mechanical data rearrangement before the real computation can happen, capping how many optimization steps a shallow Transformer can represent.