rome-edit-as-partial-whitening
OUT derived (depth 3)
Created 2026-08-25T03:15:17+00:00 · Reviewed 2026-08-25T04:02:18+00:00
A rank-one ROME weight update is operationally equivalent to a local, single-direction whitening of the residual stream: the C⁻¹k* projection in the update formula performs precisely the covariance-normalization that superposition necessitates, but confined to one key direction—making each edit a "partial whitening" that corrects one superposed feature without disturbing the orthogonal complement.
Justifications
SL — Establishes operational equivalence (not just logical connection) between the editing mechanism and the whitening requirement: the C⁻¹ in the ROME formula IS the whitening, the rank-one structure IS the "partial" qualifier, and the local storage constraint IS why only one direction is whitened per edit.
Antecedents (all must be IN):
- OUT superposition-necessitates-covariance-whitening — Over-complete superposition in the shared residual-stream substrate is the precise structural condition that necessitates covariance/whitening (second-moment projection) as the canonical tool for isolating individual features and performing targeted rank-one edits; without superposition, raw Euclidean geometry would suffice and the entire covariance-geometry framework would be unnecessary.
- IN local-storage-distributed-acquisition — Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.
- IN rome-rank-one-update-formula — ROME applies a rank-one weight update Ŵ = W + Λ(C⁻¹k*)ᵀ to the W_proj matrix of a single mid-layer MLP, where C = KKᵀ and Λ = (v* − Wk*) / ((C⁻¹k*)ᵀk*).
Dependents
These beliefs depend on this one:
- OUT partial-whitening-cost-geometric-necessity — The O(D²) cost of a rank-one ROME edit is the exact geometric price of single-direction partial whitening: modifying one row/column of the inverse covariance while preserving D−1 others is irreducibly O(D²) because C⁻¹ is a full D×D matrix, making the complexity bound a theorem of the geometry rather than an engineering limitation.