hand-constructed-weights-not-trained
IN premise — summaries/2026/08/24/shen-2023-icl-not-gd-s3-the-limiting-assumptions-in-the-study-of.md
Created 2026-08-25T02:58:32+00:00
The weight matrices in Akyürek et al. (2022) and von Oswald et al. (2023) are analytically designed (hand-constructed) and not produced by any optimization or training procedure.
Summary
Two influential papers on induction heads build their transformer weights by writing down explicit mathematical formulas rather than letting a learning algorithm find them. This matters because it means the results in those papers reflect carefully chosen mathematical properties of the architecture, not patterns discovered from data, so their findings should be read as proofs about what the structure can do rather than as evidence of what naturally emerges from training.