rome-corruption-magnitude
IN premise — summaries/2026/08/24/meng-2022-rome-s2-interventions-on-activations-for-tracing-information-flow.md
Created 2026-08-25T02:58:14+00:00
In the Causal Tracing corruption protocol, subject embeddings are perturbed by ε ~ N(0; ν) where ν = 3× the empirical standard deviation of embeddings.
Summary
When the system scrambles a model's input to figure out which parts actually drive the output, it adds noise that is deliberately large — three times the natural variation you'd see in those inputs. This sets a high bar for what counts as a genuine causal signal: if the model still behaves the same after that much disruption, the feature it's relying on is robustly important rather than just a coincidental correlation.