rome-edits-wv-value-projection

IN premise — summaries/2026/08/24/meng-2022-rome-sR-references-chunk-1.md

Created 2026-08-25T02:58:16+00:00

ROME edits W_V (the value projection) in the FFN layer, where the FFN computes f(h) = W_U·g(W_V·h + b) and h is treated as the key in the key-value memory interpretation.

Summary

ROME makes its edit by changing how the feed-forward layer extracts information from its internal memory slots — specifically the step that reads out a stored value given a lookup key — rather than changing which inputs trigger the slot or how the result is combined back into the network. This matters because it pinpoints exactly which sub-component of the layer is being rewritten, meaning the edit alters the content stored at a particular memory address without disturbing the addressing scheme or the output mixing.

Dependents

These beliefs depend on this one:

OUT residual-stream-universal-substrate — SAE (middle-layer residual stream), ROME (mid-layer MLP value projection), and Park (final-layer unembedding) all identify the residual stream at different depths as the primary locus of interpretable geometric structure.