multi-model-editing-transferability

OUT derived (depth 2)

Created 2026-08-25T03:02:14+00:00

Knowledge editing strategies (ROME-style local rank-one updates) are transferable across model families and architectures because the geometric structure of the residual stream is universal, provided that the geometric structure fully captures compositional (multi-hop) knowledge rather than only single-hop associations.

Justifications

SL — Universal geometry + universal substrate → editing should transfer. BUT the multi-hop drop (7.4% vs 40.5% on MQuAKE-CF) is direct evidence that the geometric structure does NOT capture compositional reasoning, which would break transferability. Gate is currently OUT (multi-hop drop is IN), correctly reflecting the known limitation. Flips IN if that limitation is resolved.

Antecedents (all must be IN):

IN multi-model-geometric-convergence — Both the polytope/orthogonality geometry (Park, validated on Gemma-2B and LLaMA-3-8B) and sparse feature structure (SAE, universal across architectures) converge on the finding that transformer representation spaces carry model-independent geometric invariants.
OUT residual-stream-universal-substrate — SAE (middle-layer residual stream), ROME (mid-layer MLP value projection), and Park (final-layer unembedding) all identify the residual stream at different depths as the primary locus of interpretable geometric structure.

Unless (any of these IN defeats this justification):

IN rome-gptj-mquake-cf-multi-hop-drop — ROME-edited GPT-J answers only 7.4% of MQuAKE-CF multi-hop questions, down from 40.5% before editing