superposition-increases-vulnerability-3x
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-8.md
Created 2026-08-25T02:58:00+00:00
Adversarial vulnerability increases by more than 3x as superposition forms and closely tracks the number of features per dimension (reciprocal of feature dimensionality).
Summary
When a model packs more concepts into each of its internal dimensions, it becomes far easier to craft inputs that fool it, with vulnerability more than tripling and scaling in lockstep with how many features share a single dimension. This creates a hard, predictable tradeoff: the more efficiently a model compresses its knowledge into its representation space, the more exposed it is to adversarial manipulation.