nonlinearity-enables-overcomplete-encoding
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-4.md
Created 2026-08-25T02:58:00+00:00
Introducing a single nonlinear activation (e.g., ReLU) in a 2-layer autoencoder breaks the rank-m constraint and enables representation of more features than bottleneck dimensions.
Summary
A purely linear two-layer autoencoder is hard-capped at representing only as many features as its bottleneck width, but adding even a single nonlinear activation like ReLU lifts that ceiling and lets the network encode more features than the bottleneck dimensions allow. This matters because it means the bottleneck is not a fundamental limit on what the network can represent; it is an artifact of linearity, and one small architectural change unlocks a fundamentally different representational regime.