fv-head-count-scales-with-model-size
IN premise — summaries/2026/08/24/todd-2023-function-vectors-s0-abstract-chunk-1.md
Created 2026-08-25T02:58:40+00:00
The number of attention heads in the FV-transporting set A scales with model size: GPT-J (6B) uses 10 heads, Llama 2 7B uses 20, Llama 2 13B and GPT-NeoX (20B) use 50, and Llama 2 70B uses 100.
Summary
Larger models dedicate more attention heads to the FV-transporting mechanism — it grows from 10 heads in a 6B model to 100 in a 70B model rather than staying a fixed-size module. This implies the transport capacity is an integral, scaling part of the architecture, so you cannot treat it as a constant plug-in when comparing or extrapolating behavior across different model sizes.