retrieval-hurts-recall-at-1-0.14

IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-sA-appendix.md

Created 2026-08-25T02:58:12+00:00

For the subset of questions where retrieval degrades accuracy (entity confusion), Recall@1 is approximately 0.14

Summary

When retrieval causes the model to mix up similar entities, it almost always picks the wrong answer — it only gets the top result right about 14 percent of the time. This tells the system that entity-confusion failures from retrieved context are a serious, frequent problem, not a rare edge case, and that simply adding more retrieval data is unlikely to fix it.