kandpal-2023-retrieval-inverts-accuracy-trend
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s4-methods-to-improve-rare-fact-learning.md
Created 2026-08-25T02:58:07+00:00
When retrieval-augmented context is provided, LM accuracy increases as relevant document count grows (matching human behavior), whereas closed-book accuracy decreases as rarity increases.
Summary
Giving the model a set of relevant reference documents flips its usual weakness on its head: instead of getting worse as topics become rarer (as happens when it relies solely on what it memorized), it actually gets better the more relevant material you hand it, which is how humans naturally handle unfamiliar questions. This means the system's reliability scales with the quality and quantity of retrieved context rather than with how well it has seen a topic during training, so retrieval is not just a nice-to-have but the mechanism that corrects the model's long-tail knowledge gap.
Dependents
These beliefs depend on this one:
- IN context-faithfulness-evidence — The model genuinely reads and uses provided context (not merely re-ranking retrieval) because accuracy scales with document-relevance count even when BM25 recall shows only mild dependence on the same variable.
- OUT parametric-contextual-knowledge-duality — LLMs maintain two functionally independent knowledge channels: parametric (editable via ROME's rank-one weight update) and contextual (supplied via retrieved passages that invert accuracy trends), which operate on separate mechanisms and can be independently manipulated.
- OUT retrieval-inversion-as-geometric-signature — The inversion of accuracy trends—parametric accuracy *decreases* with document count while contextual accuracy *increases*—is the operational signature of the geometric addressability boundary: the inversion occurs precisely at the subspace boundary (where the fact direction exits the rank-one addressable space), not at a frequency threshold, making the boundary a geometric rather than statistical object.