positional-encoding-shift-diagonal-matrix-property
IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-2-chunk-1.md
Created 2026-08-24T17:11:24+00:00
The sinusoidal positional encoding has the property that position shifts are expressible as diagonal matrix multiplications: f(t + Δt) = diag(f(Δt)) · f(t), enabling relative position learning as linear transformations and convolutions as linear operations.
Summary
Because the sinusoidal position encoding lets you express "shift a token forward by a few steps" as a simple matrix multiplication, the model can learn how tokens relate at different distances as a lightweight linear operation rather than memorizing absolute locations. This is a structural reason transformers handle variable-length sequences and long-range dependencies so gracefully.