icl-formal-definition-conditioning
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s1-introduction.md
Created 2026-08-25T02:58:31+00:00
In-Context Learning is formally defined as conditioning a frozen pretrained model on demonstrations in the prompt (M_Θ0(x_1 ◦ f(x_1) ◦ ... ◦ x_N ◦ f(x_N) ◦ x^t_i)), producing a distribution over the vocabulary with no parameter updates.
Summary
In-Context Learning means asking a fixed, already-trained model to make predictions by simply showing it a few examples in the input text, without changing any of its internal weights. This matters because it pins down that the model is not actually "learning" new parameters in the moment; it is just shifting its output probabilities based on the examples it sees in context, which sets a clear boundary for what the system is and is not doing.