gemma-2b-tied-embeddings-invalidate-euclidean-inner-product

IN premise — summaries/2026/08/24/park-2023-linear-representation-s15-in-the-deck-of-cards-alongside-the-queen-is-the.md

Created 2026-08-24T17:11:02+00:00

In Gemma-2B, which ties unembeddings to input token embeddings, the origin of the unembedding space carries semantic weight so raw Euclidean/cosine similarity breaks down, while the causal inner product still correctly separates concept directions; in LLaMA-2, approximately isotropic unembedding covariance makes the Euclidean inner product partially functional.

Summary

In Gemma-2B, the way input and output representations are shared means you cannot simply use standard dot-product or cosine-similarity measures to tell how close two concepts are; the "zero point" of the space is itself meaningful, which silently distorts those calculations. The practical implication is that any downstream tool relying on raw geometric similarity (clustering, nearest-neighbor search, concept alignment) will produce misleading results in Gemma-2B unless it switches to a corrected inner product, whereas the same naive approach still works reasonably well in LLaMA-2.