confidence-91-3pct-single-source-96-3pct-multi-source

IN premise — summaries/2026/08/24/xie-2024-chameleon-sloth-sA-appendix.md

Created 2026-08-25T02:59:00+00:00

LLMs show >95% confidence in 91.3% of single-source counter-answer examples and 96.3% of multi-source memory-aligned answer examples.

Summary

In nearly every case studied, the model reports extremely high confidence whether it is producing a single-source answer that pushes back on a prior claim or a multi-source answer that agrees with stored memory. This means the model's self-reported confidence is essentially flatlined near the ceiling and cannot be used to flag which outputs are more likely to be wrong or uncertain.