parametric-sufficiency-for-head-facts

OUT derived (depth 2)

Created 2026-08-25T03:10:25+00:00

Parametric MLP key-value storage is a sufficient mechanism for well-supported (head-of-distribution) knowledge, with the parametric/contextual duality providing a clean division of labor for knowledge storage and retrieval...

Justifications

SL — Local storage + duality establish sufficiency for head facts, but the 10^15-parameter estimate (a demonstrated gap/limitation) shows parametric storage is fundamentally insufficient for the long tail, gating the sufficiency claim as OUT while the scaling barrier is IN.

Antecedents (all must be IN):

IN local-storage-distributed-acquisition — Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.
OUT parametric-contextual-knowledge-duality — LLMs maintain two functionally independent knowledge channels: parametric (editable via ROME's rank-one weight update) and contextual (supplied via retrieved passages that invert accuracy trends), which operate on separate mechanisms and can be independently manipulated.

Unless (any of these IN defeats this justification):

IN kandpal-2023-scaling-estimate-10-15-params — Kandpal et al. (2023) estimate that approximately 10^15 (one quadrillion) parameters would be required to achieve competitive QA accuracy on questions with very few supporting documents.