olsson-2022-induction-heads-icl

IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-sR-references.md

Created 2026-08-25T02:58:34+00:00

Olsson et al. (2022) identified induction heads as a specific attention circuit that learns to copy prior (key, value) associations, identified as a mechanistic building block of ICL.

Summary

Olsson et al. traced a specific kind of attention layer inside transformer models — an induction head — that does one narrow job: it copies key-and-value patterns it has already seen in the input. This matters because it turns in-context learning from a mysterious emergent behavior into something concrete you can locate, trace, and test inside the network, giving the system a named, inspectable mechanism to reason about rather than a black box.