zhou-2023-realtime-qa-50-unanswerable
IN premise — summaries/2026/08/24/zhou-2023-context-faithful-prompting-s4-experiments.md
Created 2026-08-25T02:59:10+00:00
In the RealTime QA abstention task, 50 of 113 test instances were unanswerable (retrieved documents did not contain the answer) and were relabeled to the 'I don't know' option.
Summary
In this real-time QA test set, nearly half the questions (50 out of 113) had no answer available in the documents the system was given, so they were reclassified as "I don't know" cases. This means the dataset is heavily weighted toward testing whether a system can recognize its own lack of information rather than whether it can correctly answer a question, which changes how any performance numbers on it should be interpreted.