llm-lost-in-middle-wikipedia-dump-2018
IN premise — summaries/2026/08/24/liu-2023-lost-in-middle-sR-references.md
Created 2026-08-25T02:58:09+00:00
The multi-document QA retrieval corpus in Liu et al. 2023 uses a 2018 Wikipedia dump, which can contain stale facts relative to NaturalQuestions annotations.
Summary
The test documents come from a snapshot of Wikipedia taken in 2018, so some facts in those pages no longer match what the NaturalQuestions answer keys expect. This means that when a model gets a question "wrong" on this corpus, you can't always tell whether it failed because of the lost-in-the-middle attention problem or simply because the underlying text went stale before the test was built.