icl-frozen-weights-no-parameter-updates

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s3-we-call-for-more-nuanced-studies-that-maintain-parallels.md

Created 2026-08-25T02:58:32+00:00

ICL is formally defined as conditioning a pretrained model with frozen weights Θ₀ on N in-context demonstrations followed by a test input, outputting a probability distribution over vocabulary V, with no parameter updates during inference.

Summary

In-context learning is the process of feeding a few example prompt-and-response pairs into a pre-trained model and letting it generate an answer, without changing any of the model's internal weights. This matters because it means the model is not being retrained or updated; it is simply reading the examples as context and producing output in a single forward pass, which is what separates ICL from fine-tuning or gradient-based adaptation.