frobenius-norm-feature-count-proxy
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-6.md
Created 2026-08-25T02:58:00+00:00
The Frobenius norm ||W||^2_F = sum_i ||W_i||^2 is used as a basis-independent proxy for the number of learned features, since ||W_i||^2 ≈ 1 when a feature is represented and ≈ 0 otherwise.
Summary
The Frobenius norm gives the system a single, rotation-invariant number that roughly tells you how many features the network has actually learned, because each learned feature contributes about one unit to the sum while unused dimensions contribute nearly zero. This matters because it lets the system track model complexity and feature utilization without having to pick an arbitrary coordinate system, making comparisons across different parameterizations straightforward.