kandpal-2023-bm25-recall-mild-doc-dependence
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s5-related-work.md
Created 2026-08-25T02:58:07+00:00
BM25 top-k recall (k = 1–20) shows only a mild dependence on relevant document count, unlike the strong dependence observed in closed-book LM accuracy.
Summary
BM25's ability to surface the right documents stays fairly constant no matter how many relevant documents are in the pool, so the search step is not where performance breaks down as the corpus shifts. The real sensitivity to document count lives in the language model's answering step, meaning fixes should target generation rather than retrieval.
Dependents
These beliefs depend on this one:
- IN context-faithfulness-evidence — The model genuinely reads and uses provided context (not merely re-ranking retrieval) because accuracy scales with document-relevance count even when BM25 recall shows only mild dependence on the same variable.