kandpal-2023-log-linear-r2

IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-sR-references.md

Created 2026-08-25T02:58:07+00:00

Long-tail QA accuracy increases roughly linearly with log(number of model parameters) with R-squared of approximately 0.97 to 0.99 across tested model families.

Summary

Model size is the dominant, predictable lever for improving how well a system answers rare or obscure questions: each tenfold increase in parameters yields a consistent, near-linear gain in long-tail accuracy. Because this pattern holds across different model architectures with very little noise, you can rely on scaling alone to close long-tail gaps without needing a specific architectural trick.

Dependents

These beliefs depend on this one:

IN local-storage-distributed-acquisition — Factual knowledge is acquired through distributed corpus exposure (Kandpal's log-linear document-count dependence) but stored in a locally addressable MLP slot (ROME's single-layer FFN edit), revealing a two-phase knowledge pipeline.