euclidean-orthogonality-in-llama2-is-accidental
IN premise — summaries/2026/08/24/park-2023-linear-representation-s5-he-was-known-as-the-warrior.md
Created 2026-08-24T17:11:03+00:00
In LLaMA-2, causally separable concepts happen to be approximately orthogonal under the Euclidean metric due to initialization or implicit regularization favoring isotropic covariance in unembeddings, making Euclidean geometry partially functional by accident rather than reflecting genuine causal separability; the causal inner product strictly outperforms Euclidean for concept separation.
Summary
In LLaMA-2, independent concepts end up pointing in roughly perpendicular directions under the usual distance metric, but that perpendicularity is a happy accident of training initialization rather than a true reflection of how the model actually separates ideas. This means standard geometric tools like cosine similarity are only partially working by luck, and any serious effort to measure or manipulate concept independence should use a metric grounded in the model's actual causal structure to get reliable results.