knowledge-routing-architecture

OUT derived (depth 2)

Created 2026-08-25T03:03:22+00:00 · Reviewed 2026-08-25T04:28:09+00:00

The LLM implements a natural knowledge-routing architecture: well-supported facts are parametrically stored in local MLP key-value memory (head of the distribution), while rare/long-tail facts are architecturally routed through the contextual channel (retrieval, extended context windows), with the routing boundary determined by corpus document frequency rather than an arbitrary design choice.

Justifications

SL — Local storage establishes where parametric knowledge lives; duality establishes the two channels are functionally independent; externalization establishes the contextual channel is the natural home for rare facts—together they form a complete routing picture rather than three separate observations

Antecedents (all must be IN):

IN local-storage-distributed-acquisition — Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.
OUT parametric-contextual-knowledge-duality — LLMs maintain two functionally independent knowledge channels: parametric (editable via ROME's rank-one weight update) and contextual (supplied via retrieved passages that invert accuracy trends), which operate on separate mechanisms and can be independently manipulated.
IN context-externalization-principle — Rare knowledge is more efficiently stored externally (retrieval context, extended windows) than parametrically: the ~10¹⁵-parameter estimate for long-tail mastery, the 200K-token context window, and RAG-based mitigation are independent operationalizations of the same principle that context is a substitute for infeasible parametric scaling.

Dependents

These beliefs depend on this one:

OUT knowledge-routing-faithfulness-validated — The parametric/contextual two-channel knowledge architecture is a genuine computational duality rather than a surface-level re-ranking bias, because retrieval context demonstrably inverts the parametric accuracy trend (accuracy increases with document relevance for rare facts) and BM25 recall remains robust independently of parametric scaling.