In-Context Learning

104 beliefs (104 IN, 0 OUT)

• IN ahn-2024-preconditioned-gd
  Ahn et al. (2024, NeurIPS) refined the gradient descent claim to preconditioned gradient descent, addressing the gap between vanilla GD and what transformers actually compute.
• IN akyurek-2022-phase-shift-gd-to-bayes
  Akyürek et al. (2022) showed that small Transformers implement gradient descent while larger models shift toward Bayes-optimal behavior (e.g., OLS for linear regression).
• IN akyurek-2023-icl-venue
  The paper 'What Learning Algorithm Is ICL' by Akyürek et al. was published at ICLR 2023.
• IN attention-is-mathematical-dual-of-one-gradient-descent-step
  Dai et al. (ACL 2023) establish that the computation inside a Transformer attention module is the dual form of one step of gradient descent, making ICL (forward-pass meta-gradients) structurally identical to fine-tuning (backward-pass standard gradients).
• IN dai-2023-350m-model-architecture-specs
  The Dai et al. 2023 from-scratch 350M-parameter model uses 1024 hidden dimension, 4096 FFN inner dimension, 16 attention heads, 24 layers, and 1024 maximum sequence length.
• IN dai-2023-agnews-highest-learning-rate
  In Dai et al. 2023 finetuning results, the AGNews dataset received the highest learning rate (0.2) in both GPT 1.3B and GPT 2.7B model sizes, compared to much lower rates for sentiment tasks (e.g., 0.0005 for SST2 at 1.3B).
• IN dai-2023-empirical-scope-limits
  Dai et al. 2023 empirical scope is restricted to Transformer architectures, models of 2.7B parameters or fewer, and classification tasks only; results do not automatically extend to LSTMs, generation tasks, or larger models.
• IN dai-2023-gpt-meta-optimizer-gd
  Dai et al. (2023, ACL Findings) showed that GPT models implicitly perform gradient descent as 'meta-optimizers' on their prompt tokens.
• IN dai-2023-icl-meta-optimization-dual-form
  Dai et al. 2023 formalize in-context learning as a meta-optimization process via a mathematical dual form linking Transformer attention computation to steps of gradient descent, positioning ICL as implicit fine-tuning rather than pattern matching.
• IN dai-2023-icl-paper-provenance-arxiv-code
  The ICL-as-gradient-descent paper (arXiv 2212.10559v3, ACL 2023) was authored across Peking University, Tsinghua University, and Microsoft Research, with code at https://aka.ms/icl.
• IN dai-2023-icl-seed-grid-search-seven-seeds
  Dai et al. 2023 select the ICL random seed by grid search over seeds 1 through 7 per dataset per model size, acknowledging that ICL performance is sensitive to demonstration example ordering.
• IN dai-2023-lr-grid-36-candidates-sst5-exception
  Dai et al. 2023 finetuning learning rate search uses a 36-candidate grid (9 base values {1-9} × 4 scales {0.1, 0.01, 0.001, 0.0001}), with one exception: GPT 1.3B on SST5 required a finer search yielding LR = 0.00016.
• IN dai-2023-momentum-based-attention-proposed
  Dai et al. 2023 propose a momentum-based attention mechanism inspired by the meta-optimization view, which yields consistent improvements over vanilla attention across the evaluated tasks.
• IN dai-2023-optimizer-training-schedule
  Dai et al. 2023 from-scratch training uses Adam (β₁=0.9, β₂=0.98, ε=1e-6), polynomial decay LR schedule with max LR 3e-4, 500K total steps, 20K warm-up, 512K-token batch size, and gradient clip norm 2.0.
• IN dai-2023-six-classification-datasets
  Dai et al. 2023 evaluate ICL versus finetuning on six classification benchmarks: SST2, SST5, MR (Movie Reviews), Subj (Subjectivity), AGNews, and CB (CommitmentBank/NLI).
• IN dai2023-experimental-setup-gpt13b-27b-fairseq-v100-sgd-32demos
  The ICL vs. finetuning experiments used GPT 1.3B and GPT 2.7B (fairseq implementations) on NVIDIA V100 32 GB GPUs, with a maximum of 32 demonstration examples, one epoch, and SGD (not Adam) as the finetuning optimizer.
• IN dai2023-first-off-the-shelf-gpt-real-nlp-icl-gradient-descent-analysis
  Dai et al. (2023) are the first to analyze in-context learning as implicit gradient descent using off-the-shelf GPT models on real NLP classification tasks, whereas prior gradient-descent ICL results (Akyürek et al., 2022; von Oswald et al., 2022) relied on scratch-trained models on linear regression toy tasks.
• IN dai2023-icl-and-ft-both-restricted-to-kv-projections
  Both in-context learning and the compared finetuning setting restrict their effect to attention key and value projection matrices (W_K, W_V) only, not to W_Q or other layers.
• IN dai2023-kendall-icl-ft-019-021-vs-random-000
  Kendall rank correlation between ICL and finetuning attention to training tokens is 0.193 (GPT 1.3B) and 0.214 (GPT 2.7B), while Kendall correlation between ICL and random attention is 0.000 for both model sizes.
• IN dai2023-rec2ftp-icl-covers-over-85-percent-of-ft
  The Rec2FTP metric shows that ICL covers over 85% of the correct predictions achieved by finetuning over zero-shot learning, on average across the six classification tasks.
• IN fv-aie-computed-only-on-above-baseline-tasks
  AIE is computed only on tasks where the model beats the majority-label baseline given 10 ICL examples, ensuring the head set captures genuine task-relevant computation.
• IN fv-composition-outperforms-10shot-icl
  Composed FVs (v*_BD = v_B + v_D) outperform 10-shot ICL on some tasks: Last-Country-Capital achieves 0.60 ± 0.02 accuracy with composed FV vs 0.32 ± 0.02 with ICL, and Last-Capitalize-First-Letter achieves 0.95 ± 0.00 vs 0.75 ± 0.01.
• IN fv-models-and-scope-iclr-2024
  Todd et al. (ICLR 2024) demonstrate FVs across GPT-J (6B, 28L, 16 heads/layer), GPT-NeoX (20B, 44L, 64 heads), and Llama-2 7B/13B/70B, testing 40+ diverse ICL tasks.
• IN fv-qualifying-tasks-gptj-18
  18 tasks qualify for GPT-J FV extraction (those where 10-shot ICL beats the majority-label baseline).
• IN gd-accuracy-agns-8-vs-512-demos
  Gradient descent fine-tuning with 8 demos achieves 0.42 accuracy on AGNews, rising to 0.69 with 512 demos, while ICL achieves comparable accuracy with far fewer demos.
• IN gd-accuracy-rte-8-vs-512-demos
  Gradient descent fine-tuning with 8 demos achieves 0.36 accuracy on RTE, rising to 0.65 with 512 demos.
• IN gd-construction-and-trained-weights-same-loss-basin
  After correcting for scalar ambiguity in the W_K·W_Q and P·W_V products, a 50/50 linear interpolation between the analytical GD-construction weights and trained Transformer weights produces minimal loss increase, confirming both occupy the same loss basin (linear mode connectivity).
• IN gd-finetuning-modifies-weights
  Gradient descent fine-tuning modifies model parameters via the update rule Θ₁ = Θ₀ − η·∇_Θ[(1/N) Σ J(M_Θ₀(xᵢ), f(xᵢ))], after which inference is performed on a bare input with no demonstrations present.
• IN gd-learning-rates-tested-shen-2023
  Learning rates tested for the GD/cGD simulation in Shen et al. (2023) were 5e-3, 1e-3, 5e-4, and 1e-4, with training up to ~175 epochs.
• IN gd-order-stable-averages-gradients
  Standard GD is order-stable because it computes the gradient as the average loss across a batch, so reordering samples does not change the resulting parameter update.
• IN gd-order-stable-icl-order-sensitive
  Gradient descent is order-stable because it computes the mean gradient over the full batch (zero permutation sensitivity), while ICL in decoder-only Transformers is order-sensitive because each token prediction depends only on preceding tokens, making output inherently dependent on demonstration order.
• IN gd-simulating-transformer-sparsity-requirements
  Theoretical GD-simulating Transformer constructions require sparsity ratios >99.99% in W_K and W_Q, and approximately 75% in W_V, whereas real LLMs such as LLaMA-7B and GPT-J exhibit far lower (denser) weight sparsity.
• IN gd-underperforms-icl-few-demos-catches-up-512
  GD fine-tuned on 8 demonstrations achieves 0.42 accuracy on AGNews while ICL with 8 demonstrations achieves higher accuracy; with 512 demonstrations, GD reaches 0.69 accuracy on AGNews, closing the gap.
• IN gin-factorial-hmm-hidden-state-structure
  GINC uses a factorial HMM structure with 10 entities × 10 properties = 100 hidden states, 5 latent concepts, entity self-transition probability ≥ 0.9, and deterministic emissions via a memory matrix M[v,s] (p(o_t|h_t) = 1 if o_t = M[v_t, s_t]).
• IN ginc-dataset-sizes
  GINC training uses 1000 documents × 10,240 tokens and 100 validation documents × 1,024 tokens, with 2,500 prompts generated per (n,k) pair where n ∈ {0,1,2,4,8,16,32,64} and k ∈ {3,5,8,10}.
• IN ginc-entity-transition-matrix-formula
  The GINC entity transition matrix is computed as T_entity = 0.1T + 0.9I (I = 100×100 identity), enforcing ≥90% probability of staying on the same entity.
• IN ginc-memory-matrix-dimensions
  The GINC dataset uses a 10×10 memory matrix (10 entities × 10 properties) with 100 entries, where the first column is a fixed delimiter token and each entry maps to a vocabulary token.
• IN ginc-property-transition-matrix-construction
  GINC property transition matrices are generated as a convex combination of 100 random permutation matrices with softmax-distributed weights using temperature t=0.1.
• IN ginc-seeds-reporting
  All GINC experimental results are reported over 5 random seeds for variance estimation.
• IN ginc-transformer-training-config
  GINC Transformer models (GPT-2 based) use 4/12/16 layers, 12 attention heads, 768-dim embeddings, dropout 0.1, context window 1024, AdamW optimizer, batch size 8, LR 8e-4, 5 epochs, and 16-bit precision on Titan Xp GPUs.
• IN ginc-vocabulary-sizes-tested
  GINC experiments test vocabulary sizes of 50, 100, and 150, with larger vocabularies improving ICL accuracy because each hidden state is more likely to emit a distinct symbol.
• IN gpt3-lambda-longer-examples-improve-icl
  In the GPT-3 LAMBADA experiment, 5 long examples (500–600 chars) outperform 5 short examples (200–300 chars), closing ~56% of the gap to 10 independent short examples, even under train/test distribution mismatch.
• IN gptj-checkpoints-no-icl-degradation
  GPT-J checkpoints from 310k to 380k pretraining steps show no significant difference in ICL performance on AGNews, SST-2, CB, and RTE with 8 demonstrations, despite measurable parameter changes.
• IN hendel-2023-a-f-separation-accuracy
  The A/f two-stage decomposition in Hendel et al. (2023) preserves 80–90% of standard ICL accuracy, while the no-demonstration baseline T([x, →]) achieves only 10–20%.
• IN hendel-2023-core-decomposition-equation
  Hendel et al. (EMNLP 2023) formalize ICL as T([S, x]) = f(x; A(S)), where A maps demonstrations S to a task vector θ independently of query x, and f applies θ to x to produce output.
• IN hendel-hypothesis-procedure-accuracy-retention
  The Hypothesis procedure (task-vector patching) achieves approximately 80–90% of Regular ICL accuracy, while the Baseline (no demonstrations, no θ) reaches only 10–20%.
• IN icl-bayesian-ridge-lambda-sigma2-tau2
  With Gaussian prior w~N(0,τ²) and Gaussian label noise ε~N(0,σ²), the minimum-Bayes-risk estimator takes the ridge form with regularization parameter λ = σ²/τ², and ICL behaviorally matches this for all tested noise/prior ratios.
• IN icl-depth-determines-algorithm-regime
  In Akyurek et al. (ICLR 2023), transformer ICL models exhibit depth-dependent algorithmic phase transitions: ~1 layer approximates one gradient descent step, 2-4 layers approximate ridge regression, and ≥8 layers approximate OLS (λ→0).
• IN icl-empirical-validation-six-classification-tasks-attention-comparison
  The ICL-as-gradient-descent theory was empirically validated on six classification tasks by comparing predictions, attention outputs, and attention weights to query and training tokens between ICL and fine-tuning.
• IN icl-formal-definition-conditioning
  In-Context Learning is formally defined as conditioning a frozen pretrained model on demonstrations in the prompt (M_Θ0(x_1 ◦ f(x_1) ◦ ... ◦ x_N ◦ f(x_N) ◦ x^t_i)), producing a distribution over the vocabulary with no parameter updates.
• IN icl-four-primitives-each-implementable-in-single-layer
  In Akyurek et al. (2023), the four computational primitives—mov (copy), mul (matrix multiply within a column), div (element-wise divide), and aff (affine combination)—are each implementable by a single transformer decoder layer, serving as building blocks for all algorithm constructions.
• IN icl-frozen-weights-no-parameter-updates
  ICL is formally defined as conditioning a pretrained model with frozen weights Θ₀ on N in-context demonstrations followed by a test input, outputting a probability distribution over vocabulary V, with no parameter updates during inference.
• IN icl-gd-capacity-sweep-models
  The capacity invariance experiment (Section F) uses four models: GPT2-XL (1.5B), GPT-NEO (2.7B), GPT-J (6B), and LLaMA (7B) on AGNews with N=8 demonstrations.
• IN icl-gd-gap-persists-across-scales
  The performance gap between ICL and GD/cGD does not significantly close as model size increases from 1.5B to 7B parameters or as demonstration count increases from 1 to 8 on RTE.
• IN icl-gd-hypothesis-1-universal
  Hypothesis 1 in Shen et al. claims that for any Transformer weights from self-supervised pretraining and any well-defined task, ICL is algorithmically equivalent to GD (whole-model or sub-model updates).
• IN icl-gd-hypothesis-2-existence
  Hypothesis 2 in Shen et al. claims that for a given task, there exist Transformer weights (possibly hand-constructed) such that non-emergent ICL (dICL) is equivalent to GD; prior works (Akyürek et al. 2022; von Oswald et al. 2023) target this weaker claim.
• IN icl-gd-icl-is-emergent-not-trained
  ICL is an emergent behavior arising as a side effect of causal language modelling (next-word prediction) pretraining, not something explicitly targeted by a training objective.
• IN icl-gd-llama7b-probed
  The Shen et al. ICL vs. GD paper uses LLaMa-7B (pre-trained on natural data) as the primary model for empirical probing.
• IN icl-gd-order-sensitivity-discriminator
  ICL and GD exhibit different sensitivity to the order in which demonstrations are presented, which is evidence against their functional equivalence.
• IN icl-gd-paper-icml-2024
  The paper by Shen, Mishra, and Khashabi examining ICL vs. Gradient Descent equivalence was published at ICML 2024.
• IN icl-logit-via-dot-product-last-hidden-state-answer-embedding
  In the ICL classification formulation, the logit for answer y_j is computed as the dot product of the last-token hidden state M(I) with the output word embedding e_{y_j}, followed by softmax, with the predicted label selected via argmax over the restricted candidate set Y.
• IN icl-lstm-outperforms-transformer-ginc-hmm
  On GINC at vocabulary sizes 50 and 100, a 28M-parameter LSTM achieves ~95.8% in-context accuracy while a 4-layer (29M-parameter) Transformer achieves only 60–67%, attributed to structural similarity between HMMs and LSTM recurrence.
• IN icl-moment-vector-and-weight-encoded-nonlinearly-in-late-layers
  Probing experiments in Akyurek et al. (2023) show that X⊤Y is decodable around layer 7 and w_OLS around layer 12 (out of 16), and both are encoded nonlinearly (a 2-layer MLP probe outperforms a linear probe), with X⊤Y appearing earlier consistent with being an intermediate step before w is derived.
• IN icl-no-weight-update-frozen-model
  In in-context learning (ICL), the pretrained model's parameters Θ₀ remain fixed; learning occurs entirely through conditioning on concatenated demonstrations in the input context without any weight updates.
• IN icl-non-distinguishable-error-bound-o-of-1-over-k
  In the non-distinguishable regime, the excess 0-1 risk of the in-context predictor scales as O(1/k) (i.e., L_{0-1}(f_n) ≤ inf_f L_{0-1}(f) + g⁻¹(O(ε/k))), where k is the per-example length and g is the multiclass logistic calibration function.
• IN icl-order-sensitive-causal-attention
  ICL in decoder-only Transformers is order-sensitive because the causal (autoregressive) attention mask restricts each token to attend only to preceding tokens, so reordering demonstrations changes the computation.
• IN icl-prompt-ordering-sensitivity-10-40pct
  Permuting the same 4 in-context prompt examples on GINC yields 10–40% variation in accuracy, confirming that ICL accuracy is not permutation-invariant and is sensitive to example ordering.
• IN icl-ridge-behavior-requires-hidden-size-threshold
  Empirical results in Akyurek et al. (2023) show ridge-like ICL behavior requires hidden size H≥16 for d=8 and H≥32 for d=16, well below the O(d²) theoretical upper bound of 64 and 256 respectively, indicating learned implementations are more efficient than constructive proofs.
• IN icl-ridge-step-requires-od2-hidden-space
  Theorem 2 in Akyurek et al. (2023) proves that a transformer with constant depth and O(d²) hidden size can compute one Sherman-Morrison rank-one update toward the ridge-regression solution, and N exemplars require O(n) layers.
• IN icl-scale-independent-of-pretraining-loss
  In Xie et al. (2021) GINC experiments, 12-layer (85M) and 16-layer (115M) Transformers achieve identical pretraining validation loss (~1.33 at vocabulary size 50) yet differ in in-context learning accuracy, demonstrating ICL quality is not a monotonic function of perplexity alone.
• IN icl-token-format-concatenated-input-target
  In the GD-emulation construction, training tokens are concatenated as eⱼ = (xⱼ, yⱼ) in a vector of dimension N_x + N_y, the test token is initialized as (x_test, 0), and the prediction is read out as the negated y-entry of the updated test token via a final projection.
• IN icl-zero-shot-exceeds-few-shot-low-entropy
  Under low-entropy transition matrices (temperature 0.01, 12 concepts, vocabulary 100), zero-shot accuracy initially exceeds few-shot accuracy on GINC before recovering with more examples, attributed to the few-shot prompt format acting as a distractor.
• IN in-context-learning-temporary-emergent-scale
  In-context learning is temporary (unlike fine-tuning) and is an emergent property of model scale, with efficacy increasing at different rates in larger vs. smaller models
• IN in-context-learning-temporary-no-gradient-updates
  In-context learning allows LLMs to learn from prompt examples at inference time without gradient updates — it is temporary and does not produce lasting model changes unlike fine-tuning
• IN induction-heads-specific-case-of-gd-icl
  Induction heads (Olsson et al., 2022) are a specific instance of the broader gradient-descent-based in-context learning mechanism, subsumed by the equivalence between self-attention layers and GD steps shown in von Oswald et al. (2023).
• IN layernorm-degrades-gd-alignment-but-may-improve-ood
  LayerNorm degrades gradient descent alignment in self-attention layers because its averaging operation loses the input/target structural distinction needed for the dot-product construction, but may improve out-of-distribution generalization to larger-scale inputs and targets via implicit data normalization.
• IN lsa-aligns-better-with-gd-than-softmax
  Trained Transformers align more closely with the analytical GD construction under linear self-attention than under softmax attention, suggesting softmax introduces a perturbation to the ideal GD dynamics, and the effective learning rate η is encoded in the diagonal entry of P·W_V corresponding to the output dimension.
• IN lsa-gd-alignment-requires-over-2048-tasks
  Training a single LSA layer to align with the theoretical GD construction requires a fixed batch size B > 2048 tasks at the base setting (N=Nₓ=10) for near-perfect alignment; with B=128 or B=512, alignment is poor.
• IN lsa-layer-exact-gd-step-weight-construction
  A single-head linear self-attention layer implementing one GD step on L(W) = (1/2N)Σ‖Wxᵢ − yᵢ‖² uses the block construction W_K = W_Q = diag(I_x, 0), W_V = [[0,0],[W₀,−I_y]], and P = (η/N)·I, yielding the update eⱼ = (xⱼ, yⱼ) + (0, −ΔW·xⱼ).
• IN mello-8-demos-per-relation-icL-filtering
  MQuAKE dataset construction uses 8 in-context demonstration examples per relation type with GPT-J to filter out facts the model cannot recall before inclusion in the benchmark.
• IN meta-gradients-defined-as-wv-xprime-value-projections
  Meta-gradients in the ICL-as-gradient-descent framework are defined as W_V X′ (value-projection of demonstration tokens), playing the same algebraic role as the error-signal matrix E in the gradient-descent dual form.
• IN model-capacity-15b-7b-does-not-close-icl-gd-gap
  On the AGNews dataset with N=8 demonstrations, increasing model capacity from 1.5B (GPT2-XL) to 7B (LLaMA) parameters does not significantly reduce the performance gap between ICL and explicit gradient descent.
• IN momentum-based-attention-outperforms-vanilla-attention
  The momentum-based attention variant, designed by analogy with SGD-with-momentum (Polyak 1964; Sutskever et al. 2013), yields consistent performance gains over vanilla attention on both language modeling and ICL benchmarks.
• IN olsson-2022-induction-heads-icl
  Olsson et al. (2022) identified induction heads as a specific attention circuit that learns to copy prior (key, value) associations, identified as a mechanistic building block of ICL.
• IN prior-icl-gd-proofs-use-hand-constructed-weights
  The ICL≈GD equivalence proofs by Akyürek et al. (2022) and von Oswald et al. (2023) rely on hand-constructed weight matrices for which no training algorithm or procedure is specified, making them existence proofs rather than explanations of what CLM pretraining produces.
• IN real-llms-trained-via-clm-not-icl-objective
  Real-world LLMs (GPT, LLaMA) are pretrained via the causal language modeling (next-token prediction) objective; ICL emerges as a side effect of this pretraining rather than being explicitly optimized as a training target.
• IN single-attention-layer-exactly-one-gd-step
  A single linear self-attention layer with a specific weight construction is exactly equivalent to one gradient-descent step on a mean-squared-error regression loss, and a stack of k such layers equals k GD steps with iterative curvature correction (von Oswald et al., ICML 2023, arXiv:2212.07677v2).
• IN theorem1-equivalence-implies-order-sensitivity
  Theorem 1 in Shen et al. (2023) states that if an optimization algorithm A is equivalent to ICL (per Definition 1), then A must exhibit the same order sensitivity as ICL, which rules out standard GD.
• IN transformer-gd-ood-parity
  Trained Transformers and GD/GD++ exhibit nearly identical loss curves when tested on rescaled inputs (α·x, x∼U(−α,α)), rescaled teacher weights (αW, W∼N(0,I)), and entirely novel distributions (Normal, Exponential, Laplace), with the match degrading for deep non-recurrent Transformers far from the training range.
• IN two-layer-sa-circuit-copy-then-gd
  A two-layer self-attention Transformer matches one-step (not two-step) gradient descent performance because the first layer is consumed by a copying operation that merges alternating input tokens e₂ⱼ=(xⱼ) and output tokens e₂ⱼ₊₁=(0,yⱼ) into the concatenated form required for the GD computation in the second layer.
• IN w-zsl-zero-shot-effective-weights-and-delta-w-icl-decomposition
  The ICL computation decomposes into W_ZSL·q + ΔW_ICL·q where W_ZSL = W_V X (W_K X)ᵀ (zero-shot effective weights) and ΔW_ICL = Σ_i (W_V x′_i) ⊗ (W_K x′_i) (demonstration-driven update), directly mirroring F(x) = (W₀ + ΔW)x from gradient descent.
• IN xie-2021-icl-bayesian-citation
  Xie, Raghunathan, Liang, Ma (Stanford, 2021) published 'An Explanation of In-context Learning as Implicit Bayesian Inference' as arXiv:2111.02080v6.
• IN xie-2021-icl-both-architectures-ginc
  On the synthetic GINC dataset, both Transformers and LSTMs exhibit in-context learning, demonstrating the phenomenon is not specific to the attention mechanism.
• IN xie-2021-icl-error-decreases-example-length
  In Xie et al. (2021), ICL error decreases with the length of each prompt example, indicating that information in the input tokens (not just the input→output label) contributes to concept inference.
• IN xie-2021-icl-requires-long-range-coherence
  Xie et al. (2021) show that ICL emergence requires long-range coherence in pretraining documents (a shared latent concept across tokens); ablating this structure eliminates ICL on GINC.
• IN xie-2021-optimality-condition-signal-vs-mismatch
  Xie et al. (2021) prove that asymptotic ICL prediction error is optimal when the per-example signal about the latent concept θ* exceeds the error introduced by distribution mismatch between prompts and pretraining data.
• IN xie-2021-scaling-improves-icl-constant-loss
  In GINC experiments, in-context learning accuracy improves with model scale even when pretraining loss is unchanged, suggesting larger models better approximate the true posterior p(θ|prompt) rather than merely fitting the training distribution.
• IN xin-context-ginc-dataset-construction
  The GINC dataset is constructed as a uniform mixture of 5 Hidden Markov Model concepts, with 1000 pretraining documents (~10M tokens) and prompt example lengths k ∈ {3, 5, 8, 10}, evaluated at 2500 prompts per setting.
• IN xin-icl-calibration-function-formula
  The multiclass calibration function used in the ICL theory is g(δ) = ½[(1−δ)log(1−δ) + (1+δ)log(1+δ)] for δ ∈ [0,1].
• IN xin-icl-deterministic-emissions
  GINC uses a factorial HMM with deterministic emissions where p(o_t | h_t) = 1 iff o_t = M[v_t, s_t], isolating the property transition matrix as the only varying parameter.
• IN xin-icl-distinguishability-condition
  ICL convergence requires the per-token KL divergence between the true concept and any wrong concept to exceed the sum of start-distribution and delimiter error terms (Condition 1/Distinguishability).
• IN xin-icl-five-concepts-uniform-mixture
  GINC uses 5 concepts (property-transition matrices) combined in a uniform mixture, with a shared entity transition matrix across all HMMs.
• IN xin-icl-ginc-start-distribution
  The GINC start distribution is generated via softmax((u − 0.5)/t) with t=10, producing a near-uniform distribution over hidden states, sampled only once at document start.
• IN xin-icl-likelihood-ratio-convergence
  The core theoretical result states that lim_{n→∞} p(S_n | θ) / p(S_n | θ*) = 0 for θ ≠ θ*, meaning prompt likelihood concentrates on the correct concept as example count grows.