gd-underperforms-icl-few-demos-catches-up-512

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s4-icl-is-likely-not-equivalent-to-order-stable.md

Created 2026-08-25T02:58:32+00:00

GD fine-tuned on 8 demonstrations achieves 0.42 accuracy on AGNews while ICL with 8 demonstrations achieves higher accuracy; with 512 demonstrations, GD reaches 0.69 accuracy on AGNews, closing the gap.

Summary

With only a handful of example inputs, in-context learning outperforms gradient-descent fine-tuning on the AGNews classification task, but once you scale up to 512 examples the fine-tuned model closes most of that gap. This matters because it tells you that the cost of fine-tuning only pays off when you have enough labeled data to work with; for quick, low-data scenarios, just prompting the model with a few examples is the cheaper and more effective route.