zhou-memorization-ratio-reduced-35.2-to-3.0
IN premise — summaries/2026/08/24/zhou-2023-context-faithful-prompting-s1-introduction.md
Created 2026-08-25T02:59:09+00:00
Applying the proposed prompting strategies reduced the memorization ratio of text-davinci-003 from 35.2% to 3.0% on the Natural Questions dataset.
Summary
The text-davinci-003 model was leaning on memorized training data for over a third of its answers on the Natural Questions benchmark, but a change in prompting cut that dependence down to roughly one answer in thirty. This matters because it reveals the model's apparent performance was mostly pattern-matching rather than genuine understanding, and the prompting fix exposes how fragile that accuracy really was.