real-llms-trained-via-clm-not-icl-objective

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s3-a-b-in-4-5.md

Created 2026-08-25T02:58:32+00:00

Real-world LLMs (GPT, LLaMA) are pretrained via the causal language modeling (next-token prediction) objective; ICL emerges as a side effect of this pretraining rather than being explicitly optimized as a training target.

Summary

Big LLMs like GPT and LLaMA are trained with one simple goal: predict the next word in a sequence. Their ability to pick up new tasks from a few examples in a prompt is an accidental side effect of that training, not a feature that was explicitly designed or optimized, meaning it is fragile and could shift unpredictably with scale, architecture, or fine-tuning changes.