cad-pmi-theoretical-grounding
IN premise — summaries/2026/08/24/shi-2024-context-aware-decoding-s1-introduction.md
Created 2026-08-25T02:58:34+00:00
CAD's theoretical justification uses Pointwise Mutual Information between the context and the generated token, conditioned on the query and previously generated tokens, expressed as a product-of-experts distribution.
Summary
The math behind CAD boils down to measuring how strongly the context drives each newly generated token, given the query and what has already been produced, and then combining multiple such signals into one unified probability distribution. This is the theoretical backbone that positions CAD as a principled statistical method rather than an ad hoc collection of heuristics.