confidence-extraction-1000-samples-per-scenario

IN premise — summaries/2026/08/24/xie-2024-chameleon-sloth-sA-appendix.md

Created 2026-08-25T02:59:00+00:00

Confidence extraction for Llama2-7B samples 1,000 examples per scenario and computes normalized log probability over the three candidate answer tokens.

Summary

This defines the concrete recipe for measuring how confident Llama2-7B feels about an answer: it runs 1,000 example scenarios and looks only at the probability mass on three possible answer tokens, normalizing them so they add up to one. That bounded, repeated sampling gives the system a stable numeric confidence score it can actually compare across different questions, rather than relying on a single raw probability that can be inflated by the size of the vocabulary.