head-tail-geometric-divide
OUT derived (depth 4)
Created 2026-08-25T03:07:18+00:00 · Reviewed 2026-08-25T04:02:18+00:00
The parametric/contextual knowledge split is geometrically grounded rather than merely frequency-driven: head-of-distribution facts occupy individually addressable directions in the covariance-whitened feature space (enabling rank-one editing), while long-tail facts are distributed across superposed features where no single direction isolates the knowledge, making external retrieval the only faithful access mechanism.
Justifications
SL — Two antecedents establish the observed split (head→MLP, tail→context); the third supplies the geometric *explanation* for why the split exists (addressability vs. superposition), creating a causal account no single antecedent provides.
Antecedents (all must be IN):
- IN local-storage-distributed-acquisition — Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.
- IN context-externalization-principle — Rare knowledge is more efficiently stored externally (retrieval context, extended windows) than parametrically: the ~10¹⁵-parameter estimate for long-tail mastery, the 200K-token context window, and RAG-based mitigation are independent operationalizations of the same principle that context is a substitute for infeasible parametric scaling.
- OUT geometric-editing-addressability-bound — The covariance-geometry framework defines a precise and minimal addressable space for knowledge editing (rank-one updates to a single MLP value projection), but the combination of superposition and distributed corpus acquisition structurally bounds this to single-fact local corrections—edits cannot create novel multi-hop associations because the target knowledge was never locally consolidated in the first place.
Dependents
These beliefs depend on this one:
- OUT head-tail-divide-scale-validated — The head-tail geometric divide (parametric vs. contextual knowledge split) is a genuine architectural property that generalizes across model scales (334M→20B) for single-hop factual knowledge, but its extension to multi-hop compositional queries is not yet validated.
- OUT long-tail-as-geometric-addressability-failure — The long-tail knowledge problem (Kandpal's 10¹⁵-parameter estimate, 176B-model failure on rare facts) is fundamentally a geometric addressability failure rather than a data-scarcity or model-capacity issue: tail facts fail to acquire well-conditioned individual directions in the residual stream because their key vectors lie in the poorly-conditioned tail of the covariance spectrum, making them inaccessible to both parametric recall (no clean MLP slot) and rank-one editing (C⁻¹k* becomes ill-conditioned), and correctly routed to the contextual channel instead.
- OUT parametric-write-subspace-boundary — The operational boundary between parametric recall and contextual retrieval is precisely the geometric boundary of the rank-one addressable subspace: facts whose subject-key projection aligns with the locally-stored key covariance are parametrically editable, while facts outside this subspace must be externally supplied.