softmax-additive-offset-cancelled-by-two-heads
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-1.md
Created 2026-08-24T17:11:04+00:00
Softmax in attention introduces an additive offset ε relative to linear attention, making a single-head softmax layer suboptimal for gradient descent; two attention heads with sign-reversed off-diagonal W_KW_Q matrices cancel this offset, recovering PVKᵀq_j dynamics.
Summary
Softmax attention carries a hidden built-in bias that warps the gradient signal, making a single head harder to train effectively. Pairing two heads with opposite off-diagonal patterns cancels that bias out, restoring the cleaner linear-attention dynamics that gradient descent actually works best with.