icl-token-format-concatenated-input-target

IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s3-trained-transformersdo-mimic-gradient.md

Created 2026-08-24T17:11:05+00:00

In the GD-emulation construction, training tokens are concatenated as eⱼ = (xⱼ, yⱼ) in a vector of dimension N_x + N_y, the test token is initialized as (x_test, 0), and the prediction is read out as the negated y-entry of the updated test token via a final projection.

Summary

This spells out the exact data layout the gradient-descent emulation uses: each training example is a single vector with the input features and the target value glued together, a test example starts with its features filled in but a blank target slot, and the final answer is read off the target slot of the updated test vector (flipped in sign). It matters because every downstream projection and update rule depends on this split-and-concatenate convention, so getting the slot order wrong would silently corrupt the prediction readout.