cad-nqswap-construction

IN premise — summaries/2026/08/24/shi-2024-context-aware-decoding-s3-experimental-setup.md

Created 2026-08-25T02:58:35+00:00

NQ-Swap is constructed by taking NQ questions, identifying the gold named-entity answer, finding its supporting document, and replacing the gold entity with a random entity; a faithful model must answer with the replaced entity.

Summary

This describes a way to test whether a language model is genuinely reading its supporting evidence instead of relying on memorized associations. By swapping the correct answer in the source document for a different entity, the system checks if the model's answer flips to match the new evidence, which would indicate it is truly grounded in what it is given rather than pattern-matching from training.