kandpal-2023-scaling-estimate-10-15-params
IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s1-introduction.md
Created 2026-08-25T02:58:05+00:00
Kandpal et al. (2023) estimate that approximately 10^15 (one quadrillion) parameters would be required to achieve competitive QA accuracy on questions with very few supporting documents.
Summary
If a question-answering system has very few documents to draw on, it would need an absurdly large model -- around one quadrillion parameters -- just to match the accuracy you'd get with more supporting context. This sets a practical upper bound on how much you can substitute raw model size for retrieval quality, pushing the design problem toward better context provision rather than ever-larger weights.
Dependents
These beliefs depend on this one:
- IN context-externalization-principle — Rare knowledge is more efficiently stored externally (retrieval context, extended windows) than parametrically: the ~10¹⁵-parameter estimate for long-tail mastery, the 200K-token context window, and RAG-based mitigation are independent operationalizations of the same principle that context is a substitute for infeasible parametric scaling.
- OUT parametric-sufficiency-for-head-facts — Parametric MLP key-value storage is a sufficient mechanism for well-supported (head-of-distribution) knowledge, with the parametric/contextual duality providing a clean division of labor for knowledge storage and retrieval...