xie-2021-icl-implicit-bayesian-inference

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s7-discussion-and-conclusion.md

Created 2026-08-25T02:58:34+00:00

Xie et al. (2021) explain ICL as the model implicitly mapping a set of demonstrations to a latent concept (task) learned during pretraining, framed as implicit Bayesian inference.

Summary

When a language model is given a few examples in the prompt and then performs the task correctly, it is essentially recognizing which familiar task those examples point to and drawing on a statistical shortcut it already built up during pretraining. This matters because it means in-context learning is not a form of genuine new learning; the model can only handle tasks and patterns it has already absorbed in some probabilistic form, and its performance is bounded by how well it inferred the intended task from the limited examples.