dai-2023-icl-meta-optimization-dual-form
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-sR-references.md
Created 2026-08-24T17:10:54+00:00
Dai et al. 2023 formalize in-context learning as a meta-optimization process via a mathematical dual form linking Transformer attention computation to steps of gradient descent, positioning ICL as implicit fine-tuning rather than pattern matching.
Summary
Dai et al. 2023 showed that the math inside a Transformer's attention mechanism is essentially equivalent to taking a few steps of gradient descent, meaning that when a model "learns" from examples in a prompt, it is running an implicit training process in its forward pass rather than simply matching surface patterns. This reframes in-context learning as a genuine optimization event happening inside the model, which changes how we should think about what few-shot prompts can and cannot achieve and where the boundaries of model adaptability lie.