equal-loss-minima-compressed-sensing

IN premise — summaries/2026/08/24/bricken-2023-monosemanticity-chunk-6.md

Created 2026-08-25T02:57:55+00:00

MLP layers trained on compressed sensing tasks exhibit multiple equal-loss minima, some polysemantic and some monosemantic, suggesting the loss landscape alone does not determine the degree of superposition

Summary

When a neural network is trained to pack information into fewer channels, it can settle into several equally good solutions, and some of those solutions store one concept per unit while others blend many concepts into a single unit. This means the shape of the error surface by itself cannot tell you whether superposition will emerge; something beyond the loss landscape—like training order or random initialization—must be deciding which version of "good enough" the network actually lands on.