liu-2023-nq-open-queries

IN premise — summaries/2026/08/24/liu-2023-lost-in-middle-s2-multi-document-question-answering.md

Created 2026-08-25T02:58:08+00:00

The multi-document QA evaluation uses 2,655 queries from NaturalQuestions-Open with paragraph-type answers.

Summary

The evaluation is built around 2,655 open-ended questions from the Natural Questions dataset, specifically the ones that require full paragraph answers rather than a single word or number. This matters because it means the system is being tested on its ability to synthesize multi-sentence responses from multiple documents, not just retrieve a short fact.