attention-matrix-sparsity-pattern
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s6-billion-parameter-autoregressive-language.md
Created 2026-08-25T02:58:32+00:00
W^K and W^Q matrices show ~99% sparsity at all layers in LLaMa-7B and GPT-J, while W^V shows ~75% sparsity with slight decay across depth.
Summary
In both LLaMa-7B and GPT-J, the Query and Key weight matrices that determine what each token attends to are almost entirely empty at every layer, with roughly 99% of their entries being zero, while the Value matrix that carries the actual information is less sparse at about 75% and gets marginally denser in deeper layers. This means the attention mechanism is doing dramatically less computation than its matrix sizes suggest, and the model's "wiring" for deciding what to look at is far more rigid and concentrated than the wiring for what to pass along.