proposition-3-next-token-selection-mechanism
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-1.md
Created 2026-08-24T17:11:04+00:00
In Proposition 3's single-head construction, the attention mechanism selects the next token's value via KᵀW_Q e_j = p_{j+1}, where W_V contains a lower-diagonal identity I_x,off that copies y_{j+1} into token j, enabling the dot-product structure required for gradient descent.
Summary
In this construction, a single attention head is wired so that its key-query interaction outputs exactly the probability of the next token, while its value pathway simply copies that next token's value into the current position. This specific arrangement is what keeps the mechanism learnable through standard gradient descent, because it preserves the dot-product form that the optimizer depends on to update weights.