in-context-learning-temporary-no-gradient-updates

IN premise — entries/2026/06/21/wiki-Prompt_engineering.md

Created 2026-06-21T09:50:10+00:00

In-context learning allows LLMs to learn from prompt examples at inference time without gradient updates — it is temporary and does not produce lasting model changes unlike fine-tuning

Summary

When an LLM seems to "learn" from examples in a prompt, it is really just adapting its output to the pattern it sees in that single conversation — the model's underlying weights never change, so the behavior resets the moment a new session begins. This matters because it means prompt-level adaptation is disposable and cannot accumulate lasting knowledge the way fine-tuning can, so any system relying on in-context examples must treat that knowledge as session-scoped and not assume it persists.

Dependents

These beliefs depend on this one:

OUT in-context-learning-reliably-replaces-finetuning — In-context learning reliably replaces per-task fine-tuning for most downstream applications, as demonstrated by GPT-3's few-shot capabilities operating without gradient updates.