contriever-vs-bm25-popularity-dependent
IN premise — summaries/2026/08/24/mallen-2023-when-not-to-trust-s5-non-parametric-memory-complements.md
Created 2026-08-25T02:58:11+00:00
Contriever outperforms BM25 on POPQA overall, but BM25 outperforms Contriever on the least-popular entities.
Summary
Contriever looks like the better retrieval method on average, but it actually falls behind the simpler BM25 approach when the target entities are obscure or less frequently mentioned. This means the system cannot just pick one winner; the right retrieval path depends on how well-known the entity in question is, and a system that always defaults to the overall leader will silently underperform on exactly the harder, less-popular cases.