covariance-geometry-as-operational-semantic-space
OUT derived (depth 3)
Created 2026-08-25T03:05:14+00:00 · Reviewed 2026-08-25T04:28:09+00:00
The covariance/whitening geometry (second-moment matrices) is the operational definition of semantic coordinate space in LLMs: it simultaneously parameterises feature interpretation (SAE decoder space, Park polytopes), similarity evaluation (cosine→Spearman pipeline), and knowledge modification (ROME rank-one updates), and this structure converges across model families.
Justifications
SL — Three independent operational roles (interpret, evaluate, edit) all invoke the same second-moment geometry, and cross-model convergence confirms it is a structural invariant rather than an artifact of a single architecture.
Antecedents (all must be IN):
- OUT covariance-geometry-unifies-analysis-and-editing — The mathematically principled framework for both interpreting (SAE feature extraction, Park polytope analysis) and modifying (ROME rank-one edits) LLM representations is second-moment covariance geometry applied to the residual stream, since C = KKᵀ whitening defines the canonical coordinate system in which all three operations become linear algebra on the same substrate.
- IN embedding-evaluation-as-geometry-probe — The embedding evaluation pipeline (cosine similarity → Spearman correlation, validated by both SBERT and MTEB) is not an arbitrary similarity metric but a direct linear probe of the same universal feature geometry (polytopes, hierarchical orthogonality, sparse features) revealed by internal representation analysis; the observed task-specificity in MTEB scores reflects different task-specific linear projections of this shared geometric structure rather than a fundamental failure of the metric.
- IN multi-model-geometric-convergence — Both the polytope/orthogonality geometry (Park, validated on Gemma-2B and LLaMA-3-8B) and sparse feature structure (SAE, universal across architectures) converge on the finding that transformer representation spaces carry model-independent geometric invariants.
Dependents
These beliefs depend on this one:
- OUT feature-neighborhood-as-geometric-theorem-instantiation — SAE feature neighborhood structure (e.g., Golden Gate Bridge → San Francisco → California) is the concrete empirical instantiation of the covariance-geometric semantic space at the interpretable level: decoder-space proximity reflects the same subordination relations predicted by Park's orthogonality theorem, unifying interpretability with geometric theory.
- OUT geometry-ontological-status — The covariance/whitening geometry is not merely a convenient analytical tool but possesses ontological status as a genuine model-independent semantic structure, because three independent lines converge: it is the operational metric for editing and interpretation (depth-3), it is universal across architectures (depth-2), and it converges with externally-validated human-judgment metrics (depth-3).
- OUT riesz-map-as-canonical-semantic-isomorphism — The Riesz map under the causal (whitened) inner product is the unique canonical isomorphism that makes embedding and unembedding semantically identical, thereby elevating "meaning" from a model-specific activation pattern to a well-defined algebraic object in a model-independent vector space
- OUT superposition-covariance-editability-triangle — Superposition, covariance whitening, and rank-one editability form a closed logical triangle in which each property necessitates the others: over-complete superposition requires covariance separation for feature addressability, covariance separation defines the geometric space in which rank-one updates are well-defined, and the boundedness of rank-one editing confirms the addressable space is finite.