llm-output-probability-couples-embedding-and-unembedding-spaces
IN premise — summaries/2026/08/24/park-2023-linear-representation-s1-first-we-formalize-the-subspace-notion-of-linear-rep.md
Created 2026-08-24T17:11:01+00:00
The LLM next-token probability is defined as P(y|x) ∝ exp(λ(x)ᵀ γ(y)) where λ(x) ∈ Λ (embedding/context space) and γ(y) ∈ Γ (unembedding/output word space), meaning the standard dot product in the softmax is a specific inner product choice that the causal inner product replaces with a structure-respecting alternative.
Summary
The standard way an LLM scores each possible next word is really just one particular way of combining "what I've seen so far" with "what words exist to say next," and that combination is the ordinary dot product. Recognizing it as a choice rather than a fixed rule means the system can swap in a more structure-aware pairing that better respects how context and output actually relate to each other, potentially changing which words get favored and why.