mquake-eval-or-vs-and-logic
IN premise — summaries/2026/08/24/zhong-2023-mquake-sR-references-chunk-1.md
Created 2026-08-25T02:59:07+00:00
MQuAKE multi-hop accuracy uses OR logic (any generated question answered correctly counts as success) while instance-wise accuracy uses AND logic (all associated facts must be recalled).
Summary
MQuAKE scores success two different ways: one metric counts you as successful if you answer even a single generated question correctly, while the other demands you recall every associated fact without missing any. This matters because the gap between those two numbers reveals whether a model is consistently handling all pieces of a multi-hop query or just getting lucky on some, so looking at only one score can paint a misleading picture of real capability.