attention-non-symmetric

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-3.md

Created 2026-06-21T09:55:55+00:00

Attention is non-symmetric: high attention from token i to token j does not imply high attention from j to i, because W^Q and W^K are different matrices.

Dependents

These beliefs depend on this one:

• IN attention-asymmetric-stabilized-mathematical-structure — The attention mechanism has a carefully engineered mathematical structure: scaled dot-product computation with sqrt(d_k) normalization prevents gradient explosion in high dimensions, while the asymmetry (Q/K projections differ) means token-to-token relevance is inherently directional — attention from A to B does not imply attention from B to A.