kandpal-rag-mitigation

IN premise — summaries/2026/08/24/kandpal-2023-long-tail-knowledge-s0-abstract.md

Created 2026-08-25T02:58:05+00:00

Retrieval-augmentation at inference time is proposed as the practical mitigation for the long-tail knowledge problem, rather than further parametric scaling alone.

Summary

The practical fix for models missing rare or niche facts is to look up relevant information at the moment of answering, rather than relying solely on making the model bigger. This shifts the system's responsibility for accuracy away from what the model memorized during training and toward what it can retrieve and ground at inference time.

Dependents

These beliefs depend on this one:

IN context-externalization-principle — Rare knowledge is more efficiently stored externally (retrieval context, extended windows) than parametrically: the ~10¹⁵-parameter estimate for long-tail mastery, the 200K-token context window, and RAG-based mitigation are independent operationalizations of the same principle that context is a substitute for infeasible parametric scaling.