attention-is-structured-directed-computation

IN derived (depth 1)

Created 2026-06-21T11:23:58+00:00 · Reviewed 2026-06-21T14:41:08+00:00

Transformer attention is a structured directed computation — asymmetric (i→j ≠ j→i), position-dependent (requiring explicit positional encoding), and gradient-stabilized (√d_k scaling) — not a symmetric similarity metric.

Summary

Attention in a transformer is a one-way, order-sensitive lookup operation, not a simple "how alike are these two tokens" score — what token A pulls from token B is a different computation than the reverse, and it only works because positions are encoded explicitly and the math is scaled to keep training stable. This matters because any model or inference that treats attention as a symmetric similarity metric gets the wrong picture of what the network is actually computing and can therefore mispredict how changes to one token ripple through the sequence.

Justifications

SL — Three independent mechanical properties of attention collectively define it as a directed graph computation rather than pairwise similarity

Antecedents (all must be IN):

IN attention-is-non-symmetric — Attention is non-symmetric: token i attending to token j does not imply j attends to i, because the query (W^Q) and key (W^K) weight matrices are different.
IN self-attention-permutation-invariant-needs-positional-encoding — Self-attention is permutation-invariant; without positional encodings or learned positional embeddings, token order does not affect the output.
IN attention-scaled-by-sqrt-dk — Scaled dot-product attention divides by sqrt(d_k) to stabilize gradients: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) * V.

Dependents

These beliefs depend on this one:

IN attention-universality-grounded-in-structural-richness — Attention's validated universality across domains — language, protein folding, chess, reinforcement learning — is grounded in its structural computational richness: asymmetry (i attending to j does not imply j attends to i), mandatory position-dependence (requiring explicit positional encoding), and learned scaling (sqrt(d_k) stabilization) create a primitive expressive enough to serve as the sole computational mechanism for diverse sequence-processing tasks.