chameleon-gpt4-80pct-mr-popular
IN premise — summaries/2026/08/24/xie-2024-chameleon-sloth-s0-abstract-chunk-2.md
Created 2026-08-25T02:58:58+00:00
GPT-4 achieves approximately 80% memorization ratio on the most popular POPQA entities, indicating strong confirmation bias toward parametric memory for well-known facts.
Summary
When GPT-4 is asked about well-known things, it's mostly just recalling what it already has memorized in its weights rather than genuinely working through the question, and this happens about 80 percent of the time on the most common topics. The practical implication is that its strong scores on popular subjects are inflated by rote recall, so those numbers don't really tell you how well it reasons, and its answers on familiar topics are biased toward confirming what it already "knows" rather than discovering something new.