liu-2023-contriever-retriever

IN premise — summaries/2026/08/24/liu-2023-lost-in-middle-s1-introduction.md

Created 2026-08-25T02:58:08+00:00

Distractor documents for the multi-document QA task are retrieved using Contriever (fine-tuned on MS-MARCO) from Wikipedia passages of 100 tokens or fewer.

Summary

For the multi-document question-answering task, the irrelevant "distractor" passages aren't chosen at random; they are pulled from short Wikipedia snippets (under 100 tokens) using a retrieval model trained on real search queries, meaning the task is deliberately designed so the model must separate the correct passage from look-alike passages that are topically similar but wrong. This sets the difficulty floor for the benchmark and tells any downstream consumer that performance gaps likely reflect confusion over semantically close alternatives rather than basic reading failure.