retrieval-harm-rate-10pct

IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-s5-non-parametric-memory-complements.md

Created 2026-08-25T02:58:11+00:00

For approximately 10% of questions where vanilla GPT-3 was correct, retrieval augmentation caused a wrong answer, with recall@1 dropping to 0.14 in those instances versus 0.42 overall.

Summary

Adding external search results to a language model's context doesn't just fail to help sometimes—it actively flips correct answers into wrong ones about one time in ten. In those cases, the most relevant retrieved document is almost useless, which suggests the model's own internal knowledge was already sufficient and the injected context is just noise that derails it.