icl-depth-determines-algorithm-regime
IN premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-s0-abstract-chunk-1.md
Created 2026-08-24T17:10:49+00:00
In Akyurek et al. (ICLR 2023), transformer ICL models exhibit depth-dependent algorithmic phase transitions: ~1 layer approximates one gradient descent step, 2-4 layers approximate ridge regression, and ≥8 layers approximate OLS (λ→0).
Summary
A transformer's depth acts like a dial that switches the model between fundamentally different learning strategies: a single layer performs roughly one step of gradient descent, a few layers approximate regularized regression, and enough layers let it recover full unregularized regression. This matters because in-context learning is not one monolithic process — the model's computational behavior changes qualitatively as you add layers, giving us a concrete way to predict what a network will actually compute based on its architecture rather than treating it as a black box.