lsa-gd-alignment-requires-over-2048-tasks
IN premise — summaries/2026/08/24/von-oswald-2023-icl-gd-s0-ix-chunk-2.md
Created 2026-08-24T17:11:04+00:00
Training a single LSA layer to align with the theoretical GD construction requires a fixed batch size B > 2048 tasks at the base setting (N=Nₓ=10) for near-perfect alignment; with B=128 or B=512, alignment is poor.
Summary
To train a single layer of the LSA so it actually behaves like the theoretical gradient-descent target, you need to feed it over 2,048 tasks per update at the standard configuration; smaller batch sizes like 128 or 512 simply don't get the layer close enough to the ideal behavior. This sets a hard compute floor for any pipeline that depends on that layer being correctly calibrated, meaning you can't shortcut training with small batches and still trust the output.