isotropic-covariance-explains-euclidean-partial-success-in-llama2
IN premise — summaries/2026/08/24/park-2023-linear-representation-s13-every-kingdom-has-its-ruler-typically-a.md
Created 2026-08-24T17:11:02+00:00
The partial success of the Euclidean inner product in LLaMA-2 is attributed to approximately isotropic covariance of unembedding weights, likely arising from weight initialization (e.g., Xavier/Glorot) or implicit regularization during training, making unrelated directions roughly orthogonal by chance.
Summary
The dot product works reasonably well in LLaMA-2 not because it captures real semantic structure, but because the output-layer weights happen to spread evenly in all directions — a side effect of how they were initialized or regularized during training. This means the success is fragile and accidental: it depends on geometric symmetry rather than genuine understanding, and a different initialization or training setup could easily break it.