superposition-more-features-than-dimensions
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-3.md
Created 2026-08-25T02:57:59+00:00
Superposition enables a network to represent more features than it has activation dimensions by encoding features in non-orthogonal directions, effectively simulating a larger network within a fixed dimensionality budget.
Summary
A network can pack more distinct capabilities into its fixed number of internal slots by encoding them at angles to one another rather than in separate, isolated channels, much like fitting more books on a shelf by leaning them. This matters because it means the network's effective capacity exceeds what its raw size suggests, so reasoning about what a model "can do" based on its dimension count alone will underestimate its true range of behavior.