icl-moment-vector-and-weight-encoded-nonlinearly-in-late-layers

IN premise — summaries/2026/08/24/akyurek-2023-icl-algorithm-s0-abstract-chunk-2.md

Created 2026-08-24T17:10:49+00:00

Probing experiments in Akyurek et al. (2023) show that X⊤Y is decodable around layer 7 and w_OLS around layer 12 (out of 16), and both are encoded nonlinearly (a 2-layer MLP probe outperforms a linear probe), with X⊤Y appearing earlier consistent with being an intermediate step before w is derived.

Summary

The model doesn't solve regression in one linear step; it first builds up a cross-product summary of the data around a third of the way through, then uses that to derive the final weight vector about four-fifths of the way through, and at both stages the information is tangled in a way that a simple linear readout can't recover. This matters because it tells us the internal computation is genuinely nonlinear and staged, so any downstream reasoning about what the model "knows" at a given layer has to account for that hidden complexity rather than assuming a clean linear representation.