icl-gd-hypothesis-1-universal

Summary

Shen et al. make a sweeping claim that the way a pretrained Transformer "learns" from examples in its prompt is not a new trick at all but is computationally identical to standard gradient descent, whether you adjust the whole model or just a slice of it. If this holds universally across every pretrained Transformer and every task, it collapses two seemingly different learning mechanisms into one, meaning in-context learning adds no extra capacity beyond what weight updates already provide and any apparent prompt-based "learning" is secretly just gradient descent in disguise.