xie-2021-prompt-structure-delimiter

IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-s1-generate-a-start-hidden-state-hstart.md

Created 2026-08-25T02:58:55+00:00

The prompt format in Xie et al. (2021) is [x₁, y₁, o_delim, x₂, y₂, o_delim, …, xₙ, yₙ, o_delim, x_test] ∼ p_prompt, where each example is independently generated from a fresh start hidden state.

Summary

The Xie et al. prompt is simply a flat list of input-output pairs separated by marker tokens, ending with the test question. The key constraint is that the model resets its internal state between each example, so the examples act as independent demonstrations rather than a continuous context the model accumulates — which limits how the model can reason across multiple examples in the prompt.