rome-single-layer-ffn

IN premise — summaries/2026/08/24/zhong-2023-mquake-s4-mq-uake-challenges-model-editors.md

Created 2026-08-25T02:59:05+00:00

ROME localizes factual knowledge to a specific Transformer layer and updates only the feedforward network in that layer

Summary

ROME's approach rests on the observation that a Transformer stores a given fact in the feedforward connections of a single specific layer rather than spreading it across the whole network. This matters because it means a fact can be rewritten by touching only a small slice of the model, making edits cheap, targeted, and unlikely to disrupt unrelated knowledge.

Dependents

These beliefs depend on this one:

IN local-storage-distributed-acquisition — Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.