local-storage-distributed-acquisition
IN derived (depth 1)
Created 2026-08-25T03:00:40+00:00 · Reviewed 2026-08-25T04:28:09+00:00
Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.
Summary
Factual knowledge enters a model through broad, statistical exposure to large volumes of text, but ends up parked in a small, precisely addressable storage slot rather than being spread out. This two-phase separation means you can surgically correct or update a specific fact by targeting one narrow location in the network, without needing to retrain the system or reprocess the original corpus that produced the knowledge.
Justifications
This belief has 2 justifications — it is IN if any one holds.
SL — Kandpal shows acquisition is a frequency/scale phenomenon (accuracy ∝ log(params) with R²≈0.98 on rare items), while ROME shows storage is a local, single-layer phenomenon. Together they decompose the knowledge lifecycle into distributed acquisition → local consolidation.
Antecedents (all must be IN):
- IN rome-single-layer-ffn — ROME localizes factual knowledge to a specific Transformer layer and updates only the feedforward network in that layer
SL — Kandpal shows acquisition is a frequency/scale phenomenon (accuracy ∝ log(params) with R²≈0.98 on rare items), while ROME shows storage is a local, single-layer phenomenon. Together they decompose the knowledge lifecycle into distributed acquisition → local consolidation.
Antecedents (all must be IN):
- IN kandpal-2023-log-linear-r2 — Long-tail QA accuracy increases roughly linearly with log(number of model parameters) with R-squared of approximately 0.97 to 0.99 across tested model families.
Dependents
These beliefs depend on this one:
- OUT distributed-acquisition-bounds-editing-scope — Knowledge editing is fundamentally a correction mechanism (fixing mis-encoded local associations) rather than a creation mechanism, because novel factual knowledge requires the distributed corpus exposure that a single rank-one update on one MLP layer cannot replicate.
- OUT feature-level-editing-reliability — SAE-identified features can serve as interpretable, causally-grounded targets for knowledge editing—specifying edits in semantic feature space rather than raw weight matrices—because the residual stream is the universal substrate and knowledge is locally stored, contingent on feature activations being causally meaningful rather than mere statistical correlates.
- OUT head-tail-geometric-divide — The parametric/contextual knowledge split is geometrically grounded rather than merely frequency-driven: head-of-distribution facts occupy individually addressable directions in the covariance-whitened feature space (enabling rank-one editing), while long-tail facts are distributed across superposed features where no single direction isolates the knowledge, making external retrieval the only faithful access mechanism.
- OUT knowledge-lifecycle-closed-loop — The full knowledge lifecycle in LLMs—distributed corpus acquisition → local MLP key-value storage → parametric recall → rank-one editing correction—forms a closed loop unified by the covariance geometry, where each stage is addressable in the same second-moment coordinate space.
- OUT knowledge-routing-architecture — The LLM implements a natural knowledge-routing architecture: well-supported facts are parametrically stored in local MLP key-value memory (head of the distribution), while rare/long-tail facts are architecturally routed through the contextual channel (retrieval, extended context windows), with the routing boundary determined by corpus document frequency rather than an arbitrary design choice.
- OUT local-mlp-editing-principle — The MLP-as-key-value-memory structure provides a principled, minimal-intervention editing mechanism for locally-stored factual (entity-relation-object) associations, with the edit's efficacy and specificity guaranteed by the rank-one update's geometric isolation in the covariance-projected key space.
- OUT parametric-sufficiency-for-head-facts — Parametric MLP key-value storage is a sufficient mechanism for well-supported (head-of-distribution) knowledge, with the parametric/contextual duality providing a clean division of labor for knowledge storage and retrieval...
- OUT rome-edit-as-partial-whitening — A rank-one ROME weight update is operationally equivalent to a local, single-direction whitening of the residual stream: the C⁻¹k* projection in the update formula performs precisely the covariance-normalization that superposition necessitates, but confined to one key direction—making each edit a "partial whitening" that corrects one superposed feature without disturbing the orthogonal complement.
- OUT scoped-factual-knowledge-lifecycle — For single-hop factual (entity-relation-object) knowledge specifically, the full lifecycle—distributed corpus acquisition → local MLP key-value storage → parametric recall → contextual supplementation → rank-one correction—forms a coherent, validated closed loop in which each stage is geometrically well-defined and operationally distinct.