linear-model-orthogonal-symmetry
IN premise — summaries/2026/08/24/elhage-2022-toy-models-superposition-chunk-9.md
Created 2026-08-25T02:58:01+00:00
In a linear model h=Wx, x'=W^Th+b, any orthogonal matrix O yields W'=OW with identical behavior because (OW)^T(OW) = W^TW, so no privileged basis exists.
Summary
The internal structure of a linear model that encodes and then decodes data is rotationally symmetric, meaning you can re-orient the entire internal coordinate system without changing what the model does. This matters because any pattern you observe in the weights is an artifact of a particular coordinate choice rather than a fact about the data, so structural interpretations of a model's internals cannot be taken as unique or privileged.