overcomplete-basis-sparse-coding-formulation
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-11.md
Created 2026-08-25T02:57:59+00:00
The overcomplete basis formulation for solving superposition is: given activation matrix H ~ [d, m] and n underlying features (n >> m), find sparse matrix A ~ [d, n] and mixing matrix B ~ [n, m] such that H ≈ A · B.
Summary
This sets up a concrete way to recover hidden structure from data: it treats the observed patterns as sparse mixtures drawn from a pool of underlying features that is much larger than the number of dimensions you actually see. In practice, this means the system can identify the true building blocks behind a signal even when there are far more candidate features than the data dimensions would normally let you estimate, because each pattern only activates a small handful of them.