dai2023-first-off-the-shelf-gpt-real-nlp-icl-gradient-descent-analysis

IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s6-related-work.md

Created 2026-08-24T17:10:54+00:00

Dai et al. (2023) are the first to analyze in-context learning as implicit gradient descent using off-the-shelf GPT models on real NLP classification tasks, whereas prior gradient-descent ICL results (Akyürek et al., 2022; von Oswald et al., 2022) relied on scratch-trained models on linear regression toy tasks.

Summary

Dai et al. (2023) closed a key gap by showing that the "in-context learning is just gradient descent in disguise" explanation actually holds for the GPT models people use in practice, not just for tiny models solving toy math problems. This matters because it suggests the same mechanistic story governs real-world NLP behavior, giving researchers a concrete lever to predict and control how these models respond to new examples.