xin-context-ginc-dataset-construction

IN premise — summaries/2026/08/24/xie-2021-icl-bayesian-s4-simulations.md

Created 2026-08-25T02:58:56+00:00

The GINC dataset is constructed as a uniform mixture of 5 Hidden Markov Model concepts, with 1000 pretraining documents (~10M tokens) and prompt example lengths k ∈ {3, 5, 8, 10}, evaluated at 2500 prompts per setting.

Summary

This is a fixed, synthetic benchmark where the true underlying patterns are known and controlled, so it gives researchers a clean way to measure how well a model can pick up and apply hidden sequence rules from examples. The standardized setup — five equally weighted patterns, 1000 pretraining documents, prompt lengths from 3 to 10 examples, and 2500 evaluation prompts per configuration — provides a repeatable yardstick for comparing in-context learning performance across models and settings.