softmax-applied-row-wise-over-key-positions

IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-2-chunk-2.md

Created 2026-08-24T17:11:24+00:00

In the attention formula Attention(Q,K,V) = softmax(QK^T/√d_k)·V, softmax is applied row-wise (over the key positions for each query), not column-wise or globally.

Summary

Each query independently spreads its attention budget across all the keys, without any cross-talk between queries. This matters because it means one query focusing heavily on a particular key doesn't starve another query of its own ability to attend to that same key, keeping each query's attention decision fully self-contained.