icl-order-sensitive-causal-attention

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s4-icl-is-likely-not-equivalent-to-order-stable.md

Created 2026-08-25T02:58:32+00:00

ICL in decoder-only Transformers is order-sensitive because the causal (autoregressive) attention mask restricts each token to attend only to preceding tokens, so reordering demonstrations changes the computation.

Summary

In decoder-only Transformers, the order in which you present example demonstrations actually changes the model's output, because each token can only look back at what came before it, not forward. This means the arrangement of your examples is a real design lever, not a cosmetic detail; swapping two demonstrations around can produce a different result, so the model is treating your prompt as a sequence rather than a bag of interchangeable items.