eh2022-superposition-overcomplete-features
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-1.md
Created 2026-08-25T02:57:58+00:00
Superposition in neural networks means linear features are overcomplete (more features than dimensions), so the model multiplexes information into the same weight space
Summary
In a neural network, the number of useful concepts the model represents exceeds the number of weight dimensions available to store them, so the model has to layer and share the same weights across multiple concepts rather than dedicating a separate slot to each one. This is why inspecting a single weight or neuron rarely gives you one clean, isolated concept; instead, every piece of the network is a blend of several things happening at once.