gd-finetuning-modifies-weights
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s2-background.md
Created 2026-08-25T02:58:31+00:00
Gradient descent fine-tuning modifies model parameters via the update rule Θ₁ = Θ₀ − η·∇_Θ[(1/N) Σ J(M_Θ₀(xᵢ), f(xᵢ))], after which inference is performed on a bare input with no demonstrations present.
Summary
Fine-tuning works by directly adjusting a model's internal weights so it performs better on specific examples, rather than simply showing it examples at test time. The practical implication is that once fine-tuned, the model carries the adjustment internally and runs on raw input alone, without needing any demonstrations in the prompt to guide its behavior.