embedding-evaluation-as-geometry-probe
IN derived (depth 2)
Created 2026-08-25T03:03:23+00:00 · Reviewed 2026-08-25T04:28:09+00:00
The embedding evaluation pipeline (cosine similarity → Spearman correlation, validated by both SBERT and MTEB) is not an arbitrary similarity metric but a direct linear probe of the same universal feature geometry (polytopes, hierarchical orthogonality, sparse features) revealed by internal representation analysis; the observed task-specificity in MTEB scores reflects different task-specific linear projections of this shared geometric structure rather than a fundamental failure of the metric.
Summary
The way we score how well embeddings capture meaning is effectively reading the same geometric blueprint that you would find by peering inside a model's internal representations, so the numbers are grounded measurements rather than arbitrary choices. This matters because when different tasks get different scores in benchmarks, it signals that each task is probing a different angle of a shared underlying structure, not that the scoring method itself is broken.
Justifications
This belief has 3 justifications — it is IN if any one holds.
SL — Task-specificity shows features are universal while quality is task-projected; pipeline consistency shows the metric is validated across independent evaluation suites; cross-model convergence confirms the geometry is model-agnostic—each independently supports the reading that evaluation probes a real shared structure
Antecedents (all must be IN):
- OUT task-specificity-vs-feature-universality — Embedding model quality is irreducibly task-specific (no single model dominates MTEB's 8 tasks) while internal feature representations are largely model-independent (SAE features transfer across architectures), separating output-space competition from internal geometric convergence.
SL — Task-specificity shows features are universal while quality is task-projected; pipeline consistency shows the metric is validated across independent evaluation suites; cross-model convergence confirms the geometry is model-agnostic—each independently supports the reading that evaluation probes a real shared structure
Antecedents (all must be IN):
- IN embedding-evaluation-pipeline-consistency — The SBERT pipeline (NLI fine-tuning → mean pooling → cosine similarity → Spearman correlation) is the de-facto standard validated across both the original SBERT paper and the MTEB benchmark's STS evaluation protocol.
SL — Task-specificity shows features are universal while quality is task-projected; pipeline consistency shows the metric is validated across independent evaluation suites; cross-model convergence confirms the geometry is model-agnostic—each independently supports the reading that evaluation probes a real shared structure
Antecedents (all must be IN):
- IN multi-model-geometric-convergence — Both the polytope/orthogonality geometry (Park, validated on Gemma-2B and LLaMA-3-8B) and sparse feature structure (SAE, universal across architectures) converge on the finding that transformer representation spaces carry model-independent geometric invariants.
Dependents
These beliefs depend on this one:
- OUT covariance-geometry-as-operational-semantic-space — The covariance/whitening geometry (second-moment matrices) is the operational definition of semantic coordinate space in LLMs: it simultaneously parameterises feature interpretation (SAE decoder space, Park polytopes), similarity evaluation (cosine→Spearman pipeline), and knowledge modification (ROME rank-one updates), and this structure converges across model families.
- IN evaluation-as-readout-fidelity-measure — MTEB's dataset-count-weighted bias, SBERT's task-specificity gap, and the cosine→Spearman pipeline consistency are all manifestations of a single principle: embedding benchmarks measure readout fidelity (how faithfully a linear probe extracts a specific geometric direction) rather than the completeness or quality of the underlying representation space, making leaderboard rankings a statement about calibration rather than about semantics.
- IN evaluation-geometry-convergence — The convergence of embedding evaluation (cosine → Spearman correlation in SBERT/MTEB) and internal geometry analysis (covariance whitening in ROME, polytope geometry in Park) on the same second-moment structure reveals that standard benchmark evaluation is measuring the same geometric quantity that governs internal feature organization—evaluation and analysis are two readouts of one space.
- OUT evaluation-geometry-valid-independent-of-aggregation — The convergence of the cosine→Spearman evaluation pipeline across SBERT and MTEB validates the *metric itself* as a geometric probe of semantic similarity, independent of any structural bias in how per-task scores are aggregated into a final ranking.