theorem1-equivalence-implies-order-sensitivity

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s4-icl-is-likely-not-equivalent-to-order-stable.md

Created 2026-08-25T02:58:33+00:00

Theorem 1 in Shen et al. (2023) states that if an optimization algorithm A is equivalent to ICL (per Definition 1), then A must exhibit the same order sensitivity as ICL, which rules out standard GD.

Summary

Shen et al. (2023) show that any algorithm claiming to explain how language models learn from examples in their prompt must itself be sensitive to the order those examples appear, and this immediately rules out the simplest hypothesis: that in-context learning is just standard gradient descent, since gradient descent does not depend on the order in which training examples are presented.