chameleon-gpt4-popularity-mr

IN premise — summaries/2026/08/24/xie-2024-chameleon-sloth-s0-abstract-chunk-1.md

Created 2026-08-25T02:58:58+00:00

GPT-4 shows the strongest confirmation bias for high-popularity questions, reaching up to ~90% memorization ratio at 10^4–10^5 Wikipedia page views.

Summary

GPT-4 is most likely to just repeat memorized training data instead of genuinely reasoning when answering questions about very well-known topics, with up to about 90% of its answers being essentially recited from memory for questions whose subjects have tens of thousands of Wikipedia views. This matters because it means GPT-4's "thinking" degrades into parroting precisely on the subjects people ask about most, making it unreliable as a reasoning engine for popular questions.