causal-inner-product-estimable-from-unembedding-matrix
IN premise — summaries/2026/08/24/park-2023-linear-representation-s1-first-we-formalize-the-subspace-notion-of-linear-rep.md
Created 2026-08-24T17:11:01+00:00
The causal inner product in Park et al. (2023) can be estimated using the LLM's unembedding matrix alone, requiring no additional gradient-based training, and its superiority over the Euclidean dot product was validated on LLaMA-2 for 27 tested concepts including language, tense, and gender.
Summary
The causal structure of how concepts relate inside a language model can be read directly from its output (unembedding) matrix, meaning you get a causally-informed measure of concept relationships for free without any extra training or fine-tuning. This matters because it gives the system a cheap, drop-in way to distinguish causal relationships between concepts—like how tense or gender actually shape a model's reasoning—rather than relying on a plain similarity score that conflates correlation with cause.