icl-gd-step-requires-od-hidden-space

OUT premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-s0-abstract.md

Created 2026-08-24T17:10:49+00:00

Theorem 1 in Akyurek et al. (2023) proves that a transformer with constant depth and O(d) hidden size can compute one gradient descent step on the linear regression objective, and N steps require N× the layers.