mis-b-zero-iff-calibrated-not-monotonic-in-b
IN premise — summaries/2026/08/24/kalai-2023-hallucination-inevitable-s4-the-model-and-guarantees.md
Created 2026-08-24T17:10:59+00:00
With adaptive binning, Mis_b(g,p) = ‖p_{V_b(g)} − g‖_TV equals 0 if and only if g is calibrated to p for any b, and Mis_b is not monotonic in b: b=1 minimizes for g=uniform while b→∞ minimizes for g=p.
Summary
The miscalibration score is zero exactly when the model's predictions match reality, no matter how you slice the data into bins. However, the score is not simply "smaller with more bins" — with the coarsest single bin it rewards a flat, uninformative prediction, while with infinitely fine bins it rewards an exact match to the true distribution, meaning the binning resolution fundamentally changes what the metric is measuring and you cannot treat it as a stable, resolution-independent gauge of calibration quality.