lsa-layer-w0-zero-equivalent-to-nadaraya-watson-kernel-smoothing
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-1.md
Created 2026-08-24T17:11:04+00:00
A single linear self-attention layer in a Transformer block with W₀=0 computes Nadaraya-Watson kernel smoothing ŷ = Σᵢ yᵢk(xᵢ, x_test), where k(xᵢ,x_test) = m(xᵢ)ᵀm(x_test) is implicitly defined by the MLP residual connection m(x) = x + ẽ(x).
Summary
A single Transformer attention layer (with zero initial weights) turns out to be doing exactly the same computation as a classic statistical technique called kernel smoothing, where predictions are similarity-weighted averages of observed values. This matters because it gives the system a concrete, well-understood statistical interpretation of what attention is computing, letting it reason about layer behavior using established kernel-methods theory rather than treating the mechanism as a black box.