kandpal-2023-bm25-incontext-answer-verification
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s5-related-work.md
Created 2026-08-25T02:58:07+00:00
In the BM25 retrieval experiment, at least one retrieved paragraph is verified to contain the answer for every in-context example, ensuring the model learns to use provided documents.
Summary
Before testing how well a model can answer questions using retrieved documents, the researchers confirmed that every single example in their test set actually had the correct answer sitting in at least one of the provided paragraphs. This matters because it rules out a trivial failure mode — you can't blame the model for ignoring context when the context simply didn't contain the answer in the first place, so this check ensures the experiment is genuinely measuring the model's ability to use what it's given.