icl-ridge-behavior-requires-hidden-size-threshold

IN premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-s0-abstract-chunk-1.md

Created 2026-08-24T17:10:49+00:00

Empirical results in Akyurek et al. (2023) show ridge-like ICL behavior requires hidden size H≥16 for d=8 and H≥32 for d=16, well below the O(d²) theoretical upper bound of 64 and 256 respectively, indicating learned implementations are more efficient than constructive proofs.

Summary

Real transformers learn ridge-regression-style in-context behavior with far fewer hidden units than theory predicts should be necessary, meaning the models are operating more efficiently than the constructive proofs guarantee. Practically, this means we should treat theoretical lower bounds as loose upper limits on required model size, not as the true minimum, when deciding how wide a model must be to support a given learning task.