rome-rank-one-update-formula
IN premise — summaries/2026/08/24/meng-2022-rome-s2-interventions-on-activations-for-tracing-information-flow.md
Created 2026-08-25T02:58:14+00:00
ROME applies a rank-one weight update Ŵ = W + Λ(C⁻¹k*)ᵀ to the W_proj matrix of a single mid-layer MLP, where C = KKᵀ and Λ = (v* − Wk*) / ((C⁻¹k*)ᵀk*).
Summary
ROME changes a model's knowledge about one specific fact by adding a single, carefully scaled adjustment to just one layer's weights, rather than retraining the whole network. The scaling accounts for how similar that fact is to all the other facts the model already knows, so the edit is small enough to leave unrelated knowledge intact while still flipping the target association to the new value.
Dependents
These beliefs depend on this one:
- OUT rome-edit-as-partial-whitening — A rank-one ROME weight update is operationally equivalent to a local, single-direction whitening of the residual stream: the C⁻¹k* projection in the update formula performs precisely the covariance-normalization that superposition necessitates, but confined to one key direction—making each edit a "partial whitening" that corrects one superposed feature without disturbing the orthogonal complement.