privileged-basis-created-by-activation-functions
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-3.md
Created 2026-08-25T02:57:59+00:00
A privileged basis in neural networks is created by architecture-induced symmetry breaking, specifically by applying a per-neuron activation function (e.g., ReLU), which makes individual neuron directions special and encourages features to align with them.
Summary
Applying a nonlinearity like ReLU to each neuron individually breaks the symmetry that would exist in a purely linear network, making each neuron's direction a special, preferred axis in the representation space. This matters because it explains why the features a network learns tend to line up with individual neuron outputs rather than appearing in some arbitrary rotated coordinate system — the architecture itself chooses which directions count as "meaningful."