xin-icl-ginc-start-distribution
IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-sR-references.md
Created 2026-08-25T02:58:58+00:00
The GINC start distribution is generated via softmax((u − 0.5)/t) with t=10, producing a near-uniform distribution over hidden states, sampled only once at document start.
Summary
The model essentially picks its starting hidden state by rolling a nearly fair die, giving all states roughly equal odds, and does this only once per document. This means the model has no strong built-in preference for how a document begins and instead relies on the actual content to quickly steer it into the right state, which keeps the starting point flexible but places more burden on the early tokens to disambiguate.