mqa-shares-single-kv-across-all-heads

IN premise — entries/2026/06/21/wiki-Transformer_deep_learning_architecture-chunk-4.md

Created 2026-06-21T09:50:11+00:00

Multi-Query Attention (MQA) shares a single set of key/value projections across all attention heads; Grouped-Query Attention (GQA) generalizes this to groups of heads sharing KV projections.

Summary

In standard attention, every head computes its own keys and values, which is memory-heavy. MQA and GQA cut that cost by letting multiple heads reuse the same key and value computations, trading a small amount of expressiveness for much lower memory use during inference — a key trade-off for deploying large models efficiently.

Dependents

These beliefs depend on this one:

IN attention-head-sharing-is-tunable-efficiency-spectrum — Attention head parameter sharing operates along a continuous spectrum — from no sharing (standard multi-head attention) through grouped sharing (GQA, with Llama 3's 8 KV heads serving 32 query heads) to complete sharing (MQA, single KV set across all heads) — representing a tunable efficiency-quality tradeoff within the attention mechanism itself, independent of other architectural choices.