hallucination-lower-bound-is-architecture-independent
IN premise — summaries/2026/08/24/kalai-2023-hallucination-inevitable-s1-introduction.md
Created 2026-08-24T17:10:59+00:00
The Kalai-Vempala (2023) hallucination lower bound is architecture-independent: it follows purely from probability calibration and combinatorial counting of factoids, and does not depend on using transformers, attention, or any specific model class.
Summary
No matter what model architecture you build, there is a hard floor on how often it will generate confident-but-wrong facts, because the limit comes from basic probability and the sheer number of possible facts, not from any quirk of transformers or attention. In practice, this means redesigning the network cannot eliminate hallucination; the error is baked into the math of trying to be well-calibrated over a huge fact space.