w-zsl-zero-shot-effective-weights-and-delta-w-icl-decomposition

IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s2-background.md

Created 2026-08-24T17:10:53+00:00

The ICL computation decomposes into W_ZSL·q + ΔW_ICL·q where W_ZSL = W_V X (W_K X)ᵀ (zero-shot effective weights) and ΔW_ICL = Σ_i (W_V x′_i) ⊗ (W_K x′_i) (demonstration-driven update), directly mirroring F(x) = (W₀ + ΔW)x from gradient descent.

Summary

In-context learning can be understood as two clean parts stacked together: what the model already knows from its base training, plus a small correction computed directly from the examples you hand it in the prompt. This matters because it shows that the "magic" of in-context learning is structurally the same thing as one step of gradient-based weight updating, giving a concrete, predictable explanation for why demonstrations change the model's output the way they do.