multi-model-geometric-convergence
IN derived (depth 1)
Created 2026-08-25T03:00:40+00:00 · Reviewed 2026-08-25T04:28:09+00:00
Both the polytope/orthogonality geometry (Park, validated on Gemma-2B and LLaMA-3-8B) and sparse feature structure (SAE, universal across architectures) converge on the finding that transformer representation spaces carry model-independent geometric invariants.
Summary
Two independent methods for peering into how transformers organize concepts in their internal space agree on the same conclusion: the geometric layout of meaning is shaped by the structure of language itself, not by which particular model architecture you plug in. This means findings about how concepts relate to one another in a model's head can be transferred across architectures, and it points toward a shared conceptual geometry that different models are all independently converging on.
Justifications
This belief has 2 justifications — it is IN if any one holds.
SL — Park's Theorem 8 orthogonality holds across Gemma-2B (2B params) and LLaMA-3-8B; SAEs trained on GPT-2, Mistral, and Claude produce mutually more similar features than their random baselines. Two independent geometric analyses confirm architecture-invariant structure.
Antecedents (all must be IN):
- IN park-2025-validation-models-wordnet — The results were empirically validated on Gemma-2B and LLaMA-3-8B using 593 noun and 364 verb WordNet synsets (retained if containing ≥50 words in the model vocabulary).
SL — Park's Theorem 8 orthogonality holds across Gemma-2B (2B params) and LLaMA-3-8B; SAEs trained on GPT-2, Mistral, and Claude produce mutually more similar features than their random baselines. Two independent geometric analyses confirm architecture-invariant structure.
Antecedents (all must be IN):
- IN sae-universality-across-models — SAEs applied to different transformer models produce mostly similar features—more similar to each other than to their own model's neurons—suggesting features reflect data structure rather than architecture
Dependents
These beliefs depend on this one:
- OUT convergence-is-necessary-not-contingent — Cross-model geometric convergence (SAE feature similarity, Park orthogonality) is a logical necessity of superposition in a shared residual stream rather than a contingent empirical coincidence: any system that encodes d concepts in an over-complete m > d basis within a common substrate MUST produce the same covariance structure
- OUT covariance-geometry-as-operational-semantic-space — The covariance/whitening geometry (second-moment matrices) is the operational definition of semantic coordinate space in LLMs: it simultaneously parameterises feature interpretation (SAE decoder space, Park polytopes), similarity evaluation (cosine→Spearman pipeline), and knowledge modification (ROME rank-one updates), and this structure converges across model families.
- OUT cross-model-convergence-conditional-on-artifact-control — The cross-model convergence of geometric structure (polytope geometry, feature universality, orthogonality) constitutes a genuine architectural property, but the orthogonality component specifically requires set-inclusion controls to distinguish genuine hierarchical semantics from combinatorial artifacts.
- IN embedding-evaluation-as-geometry-probe — The embedding evaluation pipeline (cosine similarity → Spearman correlation, validated by both SBERT and MTEB) is not an arbitrary similarity metric but a direct linear probe of the same universal feature geometry (polytopes, hierarchical orthogonality, sparse features) revealed by internal representation analysis; the observed task-specificity in MTEB scores reflects different task-specific linear projections of this shared geometric structure rather than a fundamental failure of the metric.
- OUT multi-model-editing-transferability — Knowledge editing strategies (ROME-style local rank-one updates) are transferable across model families and architectures because the geometric structure of the residual stream is universal, provided that the geometric structure fully captures compositional (multi-hop) knowledge rather than only single-hop associations.
- OUT superposition-geometry-explains-universality — The cross-model universality of feature geometry (SAE features more similar across architectures than within, Park orthogonality validated on both Gemma and LLaMA) is a consequence of superposition: the over-complete compositional basis is determined by the shared semantic grammar of language, making geometric structure an architectural invariant rather than a model-specific artifact.
- OUT task-specificity-emerges-from-readout — Task-specificity in embedding quality is a readout phenomenon: the internal feature geometry is largely model-independent (convergent across architectures), while MTEB's no-dominant-model result arises because each task's unembedding/projection head selects a different subspace of the same shared geometric structure.