rome-mlp-key-value-interpretation

IN premise — summaries/2026/08/24/meng-2022-rome-s3-interventions-on-weights-for-understanding-factual-associati.md

Created 2026-08-25T02:58:14+00:00

In the ROME framework, W_fc in the MLP acts as the key projection and W_proj acts as the value projection, modeling the MLP as a linear associative memory where Wk ≈ v.

Summary

This treats the transformer's MLP block as a simple lookup table, where one set of weights addresses a storage slot and the other reads the stored answer back out. That reframing is what justifies ROME's surgical editing method: you can change a single fact by tweaking one entry in the table instead of retraining the entire network.

Dependents

These beliefs depend on this one:

OUT superposition-as-compositional-basis — Superposition is the fundamental compositional mechanism in LLMs: the 10–200× over-complete expansion (SAE), the key-value memory structure (ROME's W_fc/W_proj), and the direct-sum space decomposition (Park's polytope+orthogonality) are three independent geometric consequences of the same over-completeness.