mello-cot-prompt-3-demonstrations

IN premise — summaries/2026/08/24/zhong-2023-mquake-sR-references-chunk-2.md

Created 2026-08-25T02:59:08+00:00

The CoT multi-hop prompt template for MQuAKE evaluation uses 3 in-context demonstrations of 'Question → Thoughts (intermediate reasoning) → Answer' to elicit chain-of-thought decomposition.

Summary

For the MQuAKE multi-step reasoning evaluation, the system relies on showing the model three worked examples of breaking a question into visible intermediate reasoning steps before giving a final answer. This design choice matters because the structure and number of those examples directly shape how the model decomposes its thinking, which in turn affects whether the evaluation actually measures step-by-step reasoning or just pattern-matching to the demo format.