xie-2021-icl-requires-long-range-coherence
IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-s1-introduction.md
Created 2026-08-25T02:58:55+00:00
Xie et al. (2021) show that ICL emergence requires long-range coherence in pretraining documents (a shared latent concept across tokens); ablating this structure eliminates ICL on GINC.
Summary
In-context learning doesn't pop out of a model's architecture on its own; it depends on the training documents actually carrying a coherent thread that links distant parts together around a shared idea. If you break that long-range structure in the data, the model loses the ability to learn from examples given in the prompt, which means the phenomenon is data-dependent and can be controlled by how training text is organized.