llm-maintains-two-distinct-representation-spaces

IN premise — summaries/2026/08/24/park-2023-linear-representation-s2-next-we-introduce-the-notion-of-a-causal-inner-prod.md

Created 2026-08-24T17:11:02+00:00

An LLM maintains two separate d-dimensional spaces — embedding space Λ ≃ ℝᵈ (context vectors λ(x), associated with intervention) and unembedding space Γ ≃ ℝᵈ (word vectors γ(y), associated with measurement) — and the next-token probability is P(y|x) ∝ exp(λ(x)ᵀγ(y)).

Summary

An LLM doesn't blend understanding and expression into one continuous process; it keeps them as two separate vector spaces that only interact at the scoring step, where a context representation is checked against each candidate word to see how well they align. This means the system's output is always a projection onto a fixed vocabulary, and the only bridge between what it "knows" and what it can "say" is a simple alignment score between two otherwise independent representations.