kandpal-2023-eval-protocol-4shot-greedy-em
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s2-identifying-relevant-pre-training-data.md
Created 2026-08-25T02:58:06+00:00
The QA evaluation protocol uses 4-shot in-context learning, greedy decoding that stops at a newline character, and Exact Match scoring against ground-truth answer sets.
Summary
The benchmark works by showing the model four example questions, having it generate a single-line answer, and then only counting it correct if that answer is a character-for-character match to a known-correct response. This means the reported accuracy is unusually strict — no partial credit, no rephrasing tolerated — so the scores reflect a narrow, format-sensitive definition of correctness that can shift if any of those three setup choices change.