in-context-learning-temporary-emergent-scale

IN premise — entries/2026/06/21/wiki-Prompt_engineering-chunk-1.md

Created 2026-06-21T09:50:10+00:00

In-context learning is temporary (unlike fine-tuning) and is an emergent property of model scale, with efficacy increasing at different rates in larger vs. smaller models

Summary

In-context learning is a capability that only shows up because the model is large enough, and it vanishes the moment the conversation ends, so you cannot build on it the way permanent fine-tuning lets you. This means shrinking a model does not just make in-context learning slightly weaker; it hits a threshold where the ability largely disappears, and any system design that depends on prompt-based adaptation is implicitly betting on staying above that scale cutoff.