fv-head-set-sizes-by-model

IN premise — summaries/2026/08/24/todd-2023-function-vectors-s0-abstract.md

Created 2026-08-25T02:58:40+00:00

FV extraction uses a small fixed set of attention heads: 10 for GPT-J, 20 for Llama-2 7B, 50 for Llama-2 13B and GPT-NeoX 20B, and 100 for Llama-2 70B, scaled roughly proportionally to total heads.

Summary

The extraction process only looks at a small, fixed slice of each model's attention heads rather than scanning them all, and the slice grows roughly in proportion to the model's total head count. This keeps the method tractable across model sizes while implying that a tiny fraction of attention heads carries the signal the system needs.