in-context-learning-reliably-replaces-finetuning

OUT derived (depth 1)

Created 2026-06-21T10:16:20+00:00

In-context learning reliably replaces per-task fine-tuning for most downstream applications, as demonstrated by GPT-3's few-shot capabilities operating without gradient updates.

Justifications

SL — Few-shot capability is real but 40%+ accuracy shifts from prompt formatting undermine the reliability claim for production use

Antecedents (all must be IN):

IN in-context-learning-temporary-no-gradient-updates — In-context learning allows LLMs to learn from prompt examples at inference time without gradient updates — it is temporary and does not produce lasting model changes unlike fine-tuning
IN gpt3-few-shot-2020 — GPT-3 (Brown et al. 2020) demonstrated that few-shot learning emerges from scale, using 175B parameters without task-specific fine-tuning

Unless (any of these IN defeats this justification):

IN prompt-sensitivity-40-percent-accuracy-shift — LLM performance is highly sensitive to prompt design, with accuracy shifts of over 40 percentage points from minor changes such as reordering examples, and up to 76 accuracy points difference across formatting changes