attention-formula-scaled-dot-product

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-3.md

Created 2026-06-21T09:55:55+00:00

The scaled dot-product attention formula is Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) * V.

Dependents

These beliefs depend on this one:

• IN attention-asymmetric-stabilized-mathematical-structure — The attention mechanism has a carefully engineered mathematical structure: scaled dot-product computation with sqrt(d_k) normalization prevents gradient explosion in high dimensions, while the asymmetry (Q/K projections differ) means token-to-token relevance is inherently directional — attention from A to B does not imply attention from B to A.