icl-no-weight-update-frozen-model
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s2-background.md
Created 2026-08-25T02:58:31+00:00
In in-context learning (ICL), the pretrained model's parameters Θ₀ remain fixed; learning occurs entirely through conditioning on concatenated demonstrations in the input context without any weight updates.
Summary
When a large language model adapts to new examples in a prompt, it is not actually changing itself. The model's internal weights stay exactly the same as they were after training, and all the "learning" is a one-off effect of the examples sitting in the input window. This matters because it means any adaptation is temporary and context-dependent, so the system can never accumulate new skill from demonstrations the way a truly updating learner would.