document-log-likelihood-equals-sum-of-next-token-log-likelihoods

IN premise — summaries/2026/08/24/kalai-2023-hallucination-inevitable-s3-mathematical-preliminaries.md

Created 2026-08-24T17:10:59+00:00

Document log-likelihood equals the sum of next-token log-likelihoods, meaning any probability distribution over documents can be represented as a conditional next-token distribution, so hallucination is not a statistical artifact of autoregressive training.

Summary

Because a full document's probability can always be broken down into a chain of single-word predictions without losing any information, the token-by-token generation process does not introduce a statistical distortion on its own. This means that when a model produces confident but false content, the cause lies in the training data or objective rather than in the sequential, word-at-a-time architecture itself.