rome-scale-invariance-334m-to-20b
IN premise — summaries/2026/08/24/meng-2022-rome-sR-references-chunk-1.md
Created 2026-08-25T02:58:16+00:00
The two-site causal pattern (early MLP at last subject token + late attention) persists across GPT-2 Medium (334M), GPT-2 Large (774M), GPT-2 XL (1.5B), GPT-J (6B), and GPT-NeoX (20B), though peak layer indices shift.
Summary
The same two-step mechanism for tracking who-did-what in causal sentences (an early feed-forward pass on the subject, followed by a later attention step) shows up in every model tested, from the 334-million-parameter GPT-2 Medium all the way up to the 20-billion-parameter GPT-NeoX. This means the pattern is not a quirk of one architecture or size but a stable feature of how transformers are built, so any analysis or intervention targeting this mechanism can be expected to transfer across the full range of model scales.
Dependents
These beliefs depend on this one:
- OUT head-tail-divide-scale-validated — The head-tail geometric divide (parametric vs. contextual knowledge split) is a genuine architectural property that generalizes across model scales (334M→20B) for single-hop factual knowledge, but its extension to multi-hop compositional queries is not yet validated.