causal-mask-added-before-softmax-not-after

IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-2-chunk-2.md

Created 2026-08-24T17:11:25+00:00

The causal mask matrix (lower-triangular zeros, upper-triangular −∞) is added to the attention scores before softmax is applied, not after, ensuring blocked positions receive exactly zero probability mass.

Summary

In the attention mechanism, the mask that blocks future tokens is applied before the normalization step, not after, which is the only way to guarantee that blocked positions get exactly zero weight rather than just a tiny number. This matters because it is what enforces the one-way flow of information in a decoder: a token truly cannot peek ahead, and no numerical drift or rounding can leak information from the future.