causal-inner-product-outperforms-euclidean-for-concept-separability
IN premise — summaries/2026/08/24/park-2023-linear-representation-s10-the-lion-in-many-cultures-is-considered-the.md
Created 2026-08-24T17:11:01+00:00
The LOO-estimated causal inner product strictly outperforms Euclidean cosine similarity for measuring conceptual separability in LLaMA-2's unembedding space, correctly distinguishing separable from non-separable concept pairs that cosine similarity misclassifies.
Summary
When you want to know whether a language model truly distinguishes two concepts or quietly confuses them, the standard angle-based similarity measure (cosine similarity) gets some of those pairings wrong, while a causally-informed inner product correctly separates the genuinely distinct pairs from the ones the model muddles together. This matters because it gives a more reliable tool for probing what the model actually understands versus where its internal representations bleed into each other, which is critical for interpretability and for catching dangerous concept conflation.