icl-gd-icl-is-emergent-not-trained

Summary

In-context learning is not a capability the model was explicitly taught; it shows up as an unplanned side effect of simply practicing next-word prediction during pretraining. This matters because it means the behavior is tangled into the model's core learning process rather than sitting behind a clean switch, so we cannot easily tweak, isolate, or predict it without understanding how the fundamental prediction objective shaped it.