eh2022-induction-head-is-a-pair
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-1.md
Created 2026-08-25T02:57:58+00:00
An induction head consists of a pair of attention heads where the first copies a previously seen token (prefix) and the second uses that prefix to attend to a later occurrence
Summary
An induction head is not a single mechanism but a two-step team: one attention head memorizes a pattern it has seen earlier, and a partner head then searches for where that same pattern reappears later in the sequence. This matters because any analysis of inductive behavior in a model needs to track the coordinated pair, not just one head in isolation, since neither head does the real "inducing" on its own.