park-2023-softmax-affine-gauge-invariance
IN premise — summaries/2026/08/24/park-2023-linear-representation-s3-inner-product-for-language-model.md
Created 2026-08-24T17:11:03+00:00
The softmax training objective in LLMs is invariant under the affine transformation γ → Aγ + β, λ → A^{-⊤}λ, meaning the Euclidean inner product in the raw parameterization is not semantically meaningful and concept directions are identified only up to an invertible linear map A.
Summary
Because the model's predictions are unchanged if you rotate, scale, or shift its internal coordinate system in any invertible way, the specific angles and distances you measure between concept vectors in raw weight or activation space carry no inherent meaning. This matters because much interpretability work treats a particular direction as "the" direction for a concept, but that direction is really just one arbitrary representative out of a whole family of equivalent descriptions.