causal-inner-product-enforces-orthogality-of-independent-concepts

IN premise — summaries/2026/08/24/park-2023-linear-representation-s1-first-we-formalize-the-subspace-notion-of-linear-rep.md

Created 2026-08-24T17:11:01+00:00

The causal inner product in Park et al. (2023) is a non-Euclidean inner product with the property that concepts that can vary independently (e.g., gender vs. language) are represented as orthogonal vectors, and it induces a linear transformation that unifies the embedding space Λ and unembedding space Γ so both representations of a concept coincide.

Summary

This gives the model a special geometric rule: if two attributes can genuinely vary on their own (like a person's gender versus the language they speak), they are forced to be mathematically independent, so editing one won't accidentally shift the other. It also stitches together the model's "input" and "output" representation spaces into one coherent framework, so a concept has the same identity no matter which side of the model you inspect.