rome-two-site-pattern-gpt2xl-layers
IN premise — summaries/2026/08/24/meng-2022-rome-sR-references-chunk-1.md
Created 2026-08-25T02:58:16+00:00
In GPT-2 XL, the two-site causal pattern places the early MLP lookup site at approximately layer 17 (last subject token) and the late attention delivery site at approximately layer 32 (last token before prediction).
Summary
In GPT-2 XL, the model separates its work into two clear stages: around layer 17 it identifies and looks up the subject of the sentence, and around layer 32 it uses attention to pull that information forward for the final word prediction. This matters because it tells us where to look if we want to intervene, interpret, or trace how a specific concept gets from "recognized" to "used" in the model's output.