xlnet-permutation-masking-generalizes-causal
IN premise — summaries/2026-08-24/wiki-Transformer_deep_learning_architecture-chunk-2-chunk-2.md
Created 2026-08-24T17:11:25+00:00
XLNet generalizes standard causal masking by applying a random permutation P: P · M_causal · P⁻¹, allowing different tokens to have different masking patterns rather than a single fixed left-to-right order.
Summary
Instead of forcing every token to only look backward in one fixed left-to-right sequence, XLNet randomly shuffles the token order so that each token gets a different set of visible neighbors on every pass. This matters because it lets the model learn from many different "what came first" perspectives over the same sentence, rather than being locked into a single causal direction.