fv-top10-gptj-induction-heads
IN premise — summaries/2026/08/24/todd-2023-function-vectors-s2-function-vector-causal-effects-cannot-be-recovered-from-the--chunk-3.md
Created 2026-08-25T02:58:41+00:00
Three of the top-10 AIE heads in GPT-J (layers 8-1, 12-10, 24-6) are induction heads with prefix-matching scores of 0.49, 0.56, and 0.31 respectively, while several other high-AIE heads lack the prefix-matching signature.
Summary
A few of GPT-J's strongest attention heads detect when a text sequence repeats itself, but the rest of the top performers are doing something entirely different under the hood. This matters because it shows the model isn't just running one trick to get good performance; it's spreading multiple distinct pattern-recognition strategies across its layers, so any fix or analysis that assumes all top heads work the same way will miss most of what's actually happening.