dense-regime-no-privileged-basis
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-6.md
Created 2026-08-25T02:58:00+00:00
At S=0 (dense regime), there is no privileged feature basis; any set of m orthonormal directions is equally optimal, and geometric structure only emerges as sparsity increases.
Summary
When every feature is active, the system has no natural coordinate axes, so no particular decomposition of the space is more "correct" than any other. The interesting structure—which directions matter, what geometry is special—only shows up once you start turning features off, meaning sparsity is what actually gives the system its shape.