dai2023-moattn-applies-ema-to-value-vectors-not-keys-or-queries
IN premise — summaries/2026/08/24/dai-2023-icl-gradient-descent-s5-momentum-based-attention-inspired.md
Created 2026-08-24T17:10:54+00:00
Momentum-Based Attention (MoAttn) applies Exponential Moving Average to attention value vectors specifically (not keys or queries), adding Σ_{i=1}^{t-1} η^{t-i}·v_i to the standard softmax attention output, where η∈(0,1) is the decay factor.
Summary
MoAttn enriches each attention step by blending in a decaying memory of past value vectors, so the output carries a trace of what came before. Crucially, this temporal memory touches only the values (the information passed forward), leaving the keys and queries untouched, which means history shapes what the model says but not where it looks.