attention-head-sharing-is-tunable-efficiency-spectrum
IN derived (depth 1)
Created 2026-06-21T13:28:05+00:00 · Reviewed 2026-06-21T14:41:08+00:00
Attention head parameter sharing operates along a continuous spectrum — from no sharing (standard multi-head attention) through grouped sharing (GQA, with Llama 3's 8 KV heads serving 32 query heads) to complete sharing (MQA, single KV set across all heads) — representing a tunable efficiency-quality tradeoff within the attention mechanism itself, independent of other architectural choices.
Summary
How much of the attention mechanism's key-value work is shared across heads is a dial you can turn to trade a bit of model quality for real memory and speed gains, without having to change anything else about the architecture. Llama 3 sits in the middle of that dial — 8 heads serving 32 — showing this is a practical, tunable knob rather than an all-or-nothing design decision.
Justifications
SL — MHA → GQA → MQA forms a continuous parameter-sharing spectrum within attention
Antecedents (all must be IN):
- IN llama3-8-kv-heads-32-attention-heads-gqa — Llama 3 8B uses grouped-query attention with 8 key/value heads and 32 attention heads.
- IN mqa-shares-single-kv-across-all-heads — Multi-Query Attention (MQA) shares a single set of key/value projections across all attention heads; Grouped-Query Attention (GQA) generalizes this to groups of heads sharing KV projections.