attention-scaled-by-sqrt-dk

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-2.md

Created 2026-06-21T09:50:11+00:00

Scaled dot-product attention divides by sqrt(d_k) to stabilize gradients: Attention(Q,K,V) = softmax(QK^T / sqrt(d_k)) * V.

Summary

Without dividing the query-key dot products by the square root of their dimension, the values grow large enough that the softmax function saturates into a near one-hot output, and its gradients collapse to almost zero, making the model unable to learn from its attention mechanism. This single scaling step is what keeps the attention weights in a smooth, trainable range, so the rest of the network can actually adjust its parameters during training.

Dependents

These beliefs depend on this one:

IN attention-is-structured-directed-computation — Transformer attention is a structured directed computation — asymmetric (i→j ≠ j→i), position-dependent (requiring explicit positional encoding), and gradient-stabilized (√d_k scaling) — not a symmetric similarity metric.