gdpp-data-transformation-acceleration
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s3-trained-transformersdo-mimic-gradient.md
Created 2026-08-24T17:11:05+00:00
Stacking K linear self-attention layers outperforms K plain GD steps by implicitly learning a data transformation H(X) = I − γ·XXᵀ applied to inputs before each GD step (the GD++ variant), where γ is a learned scalar that adapts the update to the data geometry.
Summary
Rather than just repeating the same optimization step K times, a stack of self-attention layers learns to reshape the input data to match the problem's geometry before each step, effectively discovering a data-adaptive rule that plain gradient descent simply cannot find. This explains why the attention-based version converges faster: it is not working harder, it is quietly adjusting the problem so each individual step becomes more effective.