context-faithfulness-evidence
IN derived (depth 1)
Created 2026-08-25T03:00:40+00:00 · Reviewed 2026-08-25T04:28:09+00:00
The model genuinely reads and uses provided context (not merely re-ranking retrieval) because accuracy scales with document-relevance count even when BM25 recall shows only mild dependence on the same variable.
Summary
The model is actually reading and integrating the documents handed to it, not just leaning on the retrieval ranking as a shortcut for its own guesses. This matters because it confirms that giving the model more relevant context produces a real, compounding improvement in answer quality, so investing in better retrieval and document selection directly improves output rather than just reshuffling what the model already knew.
Justifications
SL — Multi-step argument: (1) accuracy inverts with doc count when context is provided, (2) retrieval recall does NOT show the same strong dependence. The dissociation proves the model is faithfully processing the supplied content rather than benefiting from better retrieval—evidence of genuine context faithfulness.
Antecedents (all must be IN):
- IN kandpal-2023-retrieval-inverts-accuracy-trend — When retrieval-augmented context is provided, LM accuracy increases as relevant document count grows (matching human behavior), whereas closed-book accuracy decreases as rarity increases.
- IN kandpal-2023-bm25-recall-mild-doc-dependence — BM25 top-k recall (k = 1–20) shows only a mild dependence on relevant document count, unlike the strong dependence observed in closed-book LM accuracy.
Dependents
These beliefs depend on this one:
- OUT knowledge-routing-faithfulness-validated — The parametric/contextual two-channel knowledge architecture is a genuine computational duality rather than a surface-level re-ranking bias, because retrieval context demonstrably inverts the parametric accuracy trend (accuracy increases with document relevance for rare facts) and BM25 recall remains robust independently of parametric scaling.