causal-inner-product-defined-as-inverse-covariance

IN premise — summaries/2026/08/24/park-2023-linear-representation-s15-in-the-deck-of-cards-alongside-the-queen-is-the.md

Created 2026-08-24T17:11:02+00:00

The causal inner product in Park et al. (2023) is defined as M = Cov(γ)⁻¹ (Eq. 3.3), where γ is the unembedding vector, and this construction enforces orthogality between causally separable concept directions.

Summary

In the Park et al. framework, the authors choose a specific mathematical weighting (the inverse of the covariance structure of the model's output layer) precisely so that concept directions that don't causally influence each other are forced to be geometrically perpendicular. This matters because it bakes independence into the geometry of the representation space, so the system's notion of "separate concepts" is enforced by construction rather than discovered from data.