linear-projection-max-m-directions
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-4.md
Created 2026-08-25T02:58:00+00:00
A purely linear projection from R^n to R^m can capture at most m independent directions, making the top-m principal components the optimal linear strategy.
Summary
When you squeeze high-dimensional data down to fewer dimensions using only straight-line math, you are fundamentally capped at keeping m independent axes of variation, no matter how cleverly you rotate or reweight. This is why picking the top-m principal components is the best you can do with a linear approach, and any further loss of structure is not a failure of method but a hard limit of the linear framework itself.