xin-icl-distinguishability-condition

IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-sR-references.md

Created 2026-08-25T02:58:57+00:00

ICL convergence requires the per-token KL divergence between the true concept and any wrong concept to exceed the sum of start-distribution and delimiter error terms (Condition 1/Distinguishability).

Summary

For in-context learning to actually lock onto the right concept from examples, the correct answer must be clearly distinguishable from every wrong answer by enough margin to overcome the noise in how the format starts and ends. If that gap is too small, the system simply cannot converge on the correct pattern no matter how many examples it sees.