attention-is-non-symmetric
IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-2.md
Created 2026-06-21T09:50:11+00:00
Attention is non-symmetric: token i attending to token j does not imply j attends to i, because the query (W^Q) and key (W^K) weight matrices are different.
Summary
When one word in a sentence draws focus toward another, the second word is not necessarily focusing back on the first. This one-way, directed nature of attention means the system cannot assume that information flow between two tokens is reciprocal, which affects how dependencies and context are resolved during reasoning.
Dependents
These beliefs depend on this one:
- IN attention-is-structured-directed-computation — Transformer attention is a structured directed computation — asymmetric (i→j ≠ j→i), position-dependent (requiring explicit positional encoding), and gradient-stabilized (√d_k scaling) — not a symmetric similarity metric.