dai2023-icl-and-ft-both-restricted-to-kv-projections
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s3-understanding-in-context-learning.md
Created 2026-08-24T17:10:53+00:00
Both in-context learning and the compared finetuning setting restrict their effect to attention key and value projection matrices (W_K, W_V) only, not to W_Q or other layers.
Summary
In both the in-context learning setup and the finetuning comparison, the only weights permitted to change are the key and value projection matrices inside attention heads; the query projections and every other layer are held fixed. This matters because it guarantees the two methods are competing over the same narrow slice of the model — specifically the part that governs what information gets read out of stored representations — rather than exploiting different architectural components, which makes the comparison a true test of mechanism rather than of capacity.