mquake-questions-generated-by-chatgpt

IN premise — summaries/2026/08/24/zhong-2023-mquake-sR-references-chunk-1.md

Created 2026-08-25T02:59:07+00:00

MQuAKE multi-hop questions are automatically generated by ChatGPT (gpt-3.5-turbo) from Wikidata fact chains of 2-4 triples using a 12-demoneration prompt, not human-authored.

Summary

The MQuAKE benchmark's questions were produced entirely by a language model (GPT-3.5-Turbo) working from short chains of database facts, not written by people. This matters because any systematic quirks or biases in how that model phrases questions or picks fact chains will shape the benchmark's results, so we should treat the test set as another model's output rather than a neutral human-designed evaluation.