xie-2021-icl-as-bayesian-inference
IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-s1-introduction.md
Created 2026-08-25T02:58:54+00:00
Xie et al. (2021) formalize in-context learning as the LM performing implicit Bayesian inference over a latent concept θ, where the posterior predictive p(output|prompt) = ∫ p(output|concept, prompt) p(concept|prompt) d(concept) is the core mathematical object.
Summary
When a language model handles a prompt, it is effectively doing statistical guesswork about what underlying topic or intent the user has in mind, and then answering based on that inferred topic rather than just copying surface patterns. This gives the system a principled, mathematically grounded explanation for why in-context learning works: the model behaves like a rational reasoner updating its estimate of the situation and then predicting what comes next.