superposition-dense-regime-pca-behavior

IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-2.md

Created 2026-08-25T02:57:58+00:00

In the dense feature activation regime, the toy model learns an orthogonal basis representing only the top-k most important features, analogous to PCA, leaving remaining features unrepresented.

Summary

When many features are active at once, the model essentially reduces everything down to just the few most dominant signals and discards the rest, behaving like a principal-component analysis. This means the system is making a lossy compression in crowded scenarios, so any feature that isn't among the top few simply gets dropped from the model's internal representation, which can lead to missed information if a "minor" feature turns out to matter later.