information-content-defined-as-negative-log-probability
IN premise — summaries/2026/08/24/wiki-Semantic_similarity-chunk-2.md
Created 2026-08-25T02:58:53+00:00
Information Content of a concept x is defined as IC(x) = −log P(x), meaning more specific (less probable) concepts have higher information content.
Summary
This sets up a simple rule for how much "information" a concept carries: the rarer or more specific something is, the more it tells you, because it rules out more possibilities. In practice, it means the system can weight specific, low-probability details more heavily than vague, high-probability ones when reasoning or measuring surprise.