rome-gptj-mquake-cf-multi-hop-drop
IN premise — summaries/2026/08/24/zhong-2023-mquake-s1-introduction.md
Created 2026-08-25T02:59:04+00:00
ROME-edited GPT-J answers only 7.4% of MQuAKE-CF multi-hop questions, down from 40.5% before editing
Summary
ROME's knowledge-editing procedure, meant to surgically update a single fact in GPT-J, ends up crippling the model's ability to chain together multiple related facts, dropping accuracy on multi-step questions from about 40% to under 8%. This suggests that ROME's edits ripple far beyond the targeted fact, breaking the relational structure the model needs for connected reasoning.
Dependents
These beliefs depend on this one:
- OUT head-tail-divide-scale-validated — The head-tail geometric divide (parametric vs. contextual knowledge split) is a genuine architectural property that generalizes across model scales (334M→20B) for single-hop factual knowledge, but its extension to multi-hop compositional queries is not yet validated.
- OUT inversion-and-drop-as-dual-boundary — Kandpal's retrieval inversion (parametric accuracy *decreases* with document count while contextual accuracy *increases*) and ROME's multi-hop drop (40.5%→7.4% after editing) are the same geometric boundary observed from the read and write sides respectively: the rank-one addressable subspace fills the head directions parametrically but is silent on the tail, and the context window fills exactly those missing tail directions.
- OUT multi-hop-drop-as-rank-one-boundary-signature — The dramatic 40.5%→7.4% multi-hop accuracy collapse after ROME editing is the operational fingerprint of the rank-one write boundary: multi-hop reasoning requires *composition* of multiple whitened directions, but the covariance-whitened subspace structurally supports only single-direction writes, making multi-hop parametric recall geometrically unreachable.
- OUT multi-model-editing-transferability — Knowledge editing strategies (ROME-style local rank-one updates) are transferable across model families and architectures because the geometric structure of the residual stream is universal, provided that the geometric structure fully captures compositional (multi-hop) knowledge rather than only single-hop associations.
- OUT scoped-factual-knowledge-lifecycle — For single-hop factual (entity-relation-object) knowledge specifically, the full lifecycle—distributed corpus acquisition → local MLP key-value storage → parametric recall → contextual supplementation → rank-one correction—forms a coherent, validated closed loop in which each stage is geometrically well-defined and operationally distinct.
- OUT single-fact-edit-efficacy — ROME's high efficacy and specificity on individual factual associations holds as a reliable operational claim, but is scoped to single-hop entity-relation edits.
- OUT three-tier-architecture-completeness — The three-tier memory architecture (broad superposition read, narrow parametric write, unbounded context write) constitutes a complete and faithful model of LLM knowledge management