park-2023-causal-inner-product-closed-form

IN premise — summaries/2026/08/24/park-2023-linear-representation-s3-inner-product-for-language-model.md

Created 2026-08-24T17:11:03+00:00

The causal inner product in Park et al. (2023) has the closed-form expression ⟨γ̄, γ̄'⟩_C = γ̄^⊤ Cov(γ)^{-1} γ̄', where γ is the unembedding vector of a vocabulary-uniform word, making the inverse vocabulary-covariance matrix serve as the metric (analogous to Mahalanobis distance).

Summary

Park et al. show that the causal inner product between word embeddings can be computed directly using a formula that weighs differences by the inverse of how vocabulary features co-vary, essentially treating the vocabulary's statistical structure as a built-in ruler for causal relatedness. This matters because it turns a potentially expensive sampling or simulation step into a simple matrix multiply, giving the system a fast, closed-form way to compare causal influence between words without approximating the geometry numerically.

Dependents

These beliefs depend on this one:

IN covariance-geometry-as-canonical-tool — Independent lines of work (ROME's key-space projection and Park's unembedding whitening) converge on using empirical second-moment matrices to define the "correct" inner product for reasoning about transformer representations.