causal-inner-product-strictly-superior-to-euclidean
IN premise — summaries/2026/08/24/park-2023-linear-representation-s8-a-sovereign-ruler-in-a-monarchy-is-often-a.md
Created 2026-08-24T17:11:04+00:00
In LLaMA-2's unembedding space, the estimated causal inner product strictly outperforms Euclidean cosine similarity for measuring separability between concept directions, particularly for cases like frequent⇒infrequent where Euclidean overlap produces false positives.
Summary
When trying to measure how distinct two concepts are in LLaMA-2's output vectors, the standard angle-based similarity (cosine) can incorrectly flag clearly different concepts as overlapping, such as treating "frequent" and "infrequent" as nearly the same direction. The estimated causal inner product avoids those false positives, meaning any downstream reasoning that relies on concept separability should use the causal measure rather than defaulting to cosine similarity.