gemma-2b-tied-weights-break-euclidean-geometry
IN premise — summaries/2026/08/24/park-2023-linear-representation-s5-king-head-queen-woman-em.md
Created 2026-08-24T17:11:03+00:00
Gemma-2B ties its unembedding matrix to its token embedding matrix, causing the Euclidean inner product to fail as a semantic metric for concept separation, while the causal inner product (M = Cov(γ)⁻¹) still correctly separates causally independent concepts—demonstrating that architectural choice (tied vs. untied weights) fundamentally affects whether Euclidean geometry is meaningful.
Summary
Because Gemma-2B reuses a single weight matrix for both encoding tokens into vectors and decoding vectors back into tokens, the usual angle-and-distance shortcuts in its vector space no longer reliably reflect real differences between concepts. The practical takeaway is that the model builders' choice to share those weights changes the geometry of the space itself, so anyone interpreting the model's internal representations needs a dependency-aware measure rather than naive geometry to correctly tell apart concepts that are truly independent.