superposition-sharp-phase-transition
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-11.md
Created 2026-08-25T02:57:59+00:00
Superposition and polysemanticity exhibit a sharp phase transition (not a gradual continuum): there exists a regime boundary beyond which superposition is entirely absent for any given model architecture and sparsity level.
Summary
For any given model design and sparsity setting, there is a hard cutoff: beyond that point, neurons stop sharing roles entirely, and the overlap just vanishes rather than fading out slowly. This matters because it means you cannot dial down feature-sharing gradually; you either have it or you don't, which changes how compression, interpretation, and architecture choices should be made.