icl-empirical-validation-six-classification-tasks-attention-comparison
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s1-introduction.md
Created 2026-08-24T17:10:53+00:00
The ICL-as-gradient-descent theory was empirically validated on six classification tasks by comparing predictions, attention outputs, and attention weights to query and training tokens between ICL and fine-tuning.
Summary
This is direct evidence that when a large language model "learns" from examples dropped into its prompt, the internal math looks strikingly similar to what happens during ordinary fine-tuning training, as confirmed across six classification benchmarks. It matters because it grounds in-context learning in the well-understood mechanics of gradient descent, meaning we can predict, debug, and explain prompt-based learning behavior using tools and intuitions we already have from standard model training.