self-attention-permutation-invariant-needs-positional-encoding

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture.md

Created 2026-06-21T09:50:10+00:00

Self-attention is permutation-invariant; without positional encodings or learned positional embeddings, token order does not affect the output.

Summary

Self-attention, the core computation in transformer models, treats its input as an unordered set, so on its own it cannot tell the difference between "the dog bit the man" and "the man bit the dog." This is why every practical transformer must layer in separate positional signals; without them, all word-order information is lost and the model becomes blind to grammar and sequence.

Dependents

These beliefs depend on this one:

IN attention-is-structured-directed-computation — Transformer attention is a structured directed computation — asymmetric (i→j ≠ j→i), position-dependent (requiring explicit positional encoding), and gradient-stabilized (√d_k scaling) — not a symmetric similarity metric.
IN positional-encoding-is-a-diverse-unsettled-design-choice — Multiple competing positional encoding strategies (absolute sinusoidal, RoPE, ALiBi) exist because self-attention is inherently permutation-invariant — each approach trades off different properties.