mquake-recallability-filter-8shot
IN premise — summaries/2026/08/24/zhong-2023-mquake-s2-problem-definition.md
Created 2026-08-25T02:59:05+00:00
The MQuAKE construction pipeline discards any chain containing a fact that GPT-J cannot recall via 8-shot in-context learning
Summary
The MQuAKE dataset only keeps question chains where every underlying fact can be pulled out of GPT-J from just eight examples, so any chain touching a harder or rarer fact gets thrown out entirely. In practice, this means the dataset's coverage is quietly shaped by one model's recall limits, and any downstream work built on it inherits a bias toward facts that GPT-J already finds easy.