rome-two-site-causal-architecture

IN premise — summaries/2026/08/24/meng-2022-rome-sR-references.md

Created 2026-08-25T02:58:17+00:00

Causal Tracing identifies two critical sites for factual recall in GPT-2 XL: an early site (~layers 15–17) at the last subject token where MLP lookups are decisive, and a late site (~layer 32) at the last token before prediction where attention modules dominate.

Summary

Factual recall in GPT-2 XL happens in two distinct stages: the model first retrieves the relevant fact in the middle of the network (around layers 15-17) through its feedforward components, and then integrates that fact into the final output near the end (around layer 32) through its attention mechanism. This two-site architecture matters because it tells us that if you want to intervene on what the model says, monitor its reasoning, or edit its factual behavior, you now know exactly which layers and which token positions are the critical gates to target rather than treating the network as a black box.