kandpal-2023-human-inverse-trend-rare-facts
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s3-lm-accuracy-depends-on-relevant.md
Created 2026-08-25T02:58:06+00:00
Humans show the inverse trend to LLMs: human accuracy is highest on questions with few relevant pre-training documents, suggesting the LM difficulty is due to frequency of exposure rather than question difficulty.
Summary
When a fact is rare in the training data, people actually handle those questions better than language models do, which points to the real bottleneck: the model simply hasn't seen those facts enough times to learn them well. This matters because it means improving performance on rare or niche topics is a data-exposure problem, not a reasoning-difficulty problem.