eh2022-sparsity-prerequisite-for-superposition
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-10.md
Created 2026-08-25T02:57:58+00:00
Sparsity of features (active on only a small subset of inputs) is the prerequisite that makes superposition beneficial; if features were dense, the dimensionality savings would vanish
Summary
Superposition, the idea of packing more features into fewer dimensions, only works because each feature is active on a small fraction of inputs rather than most of them. If features were broadly active, they would constantly interfere with one another and the space-saving benefit that makes superposition worthwhile would disappear entirely.