attention-scaling-factor-sqrt-dk

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-3.md

Created 2026-06-21T09:55:55+00:00

Attention scores are divided by sqrt(d_k) (not d_k) to stabilize gradients during training.

Dependents

These beliefs depend on this one:

• IN attention-asymmetric-stabilized-mathematical-structure — The attention mechanism has a carefully engineered mathematical structure: scaled dot-product computation with sqrt(d_k) normalization prevents gradient explosion in high dimensions, while the asymmetry (Q/K projections differ) means token-to-token relevance is inherently directional — attention from A to B does not imply attention from B to A.