llama3-8-kv-heads-32-attention-heads-gqa

IN premise — entries/2026/06/21/wiki-LLaMA.md

Created 2026-06-21T09:50:09+00:00

Llama 3 8B uses grouped-query attention with 8 key/value heads and 32 attention heads.

Summary

This records that the model shares key and value computations across groups of four query heads rather than computing them separately for all 32 heads, a design choice that shrinks the memory footprint during inference. The practical upshot is that the model runs faster and fits in less RAM than a full multi-head design with the same query capacity, which is part of why an 8B-parameter model is viable on consumer hardware.

Dependents

These beliefs depend on this one:

IN attention-head-sharing-is-tunable-efficiency-spectrum — Attention head parameter sharing operates along a continuous spectrum — from no sharing (standard multi-head attention) through grouped sharing (GQA, with Llama 3's 8 KV heads serving 32 query heads) to complete sharing (MQA, single KV set across all heads) — representing a tunable efficiency-quality tradeoff within the attention mechanism itself, independent of other architectural choices.
IN post-transformer-refinements-stabilize-without-changing-fundamentals — Post-2017 architectural refinements — Pre-LN normalization (eliminating warmup), SwiGLU activation, RMSNorm, grouped-query attention — all improve training stability and inference efficiency without altering the fundamental self-attention mechanism, representing evolutionary optimization of a settled architecture.