dai-2023-gpt-meta-optimizer-gd

Summary

GPT models, when processing a prompt, are secretly running the same optimization loop (gradient descent) that was used to train them, but pointed at the prompt's own words instead of the model's weights. This matters because it gives a concrete mechanical explanation for in-context learning: the model isn't doing some mysterious one-shot reasoning trick, it is doing tiny in-flight learning on the fly, which helps explain both why prompt engineering works and where its limits will show up.