evaluation-geometry-valid-independent-of-aggregation

OUT derived (depth 4)

Created 2026-08-25T03:48:00+00:00

The convergence of the cosine→Spearman evaluation pipeline across SBERT and MTEB validates the *metric itself* as a geometric probe of semantic similarity, independent of any structural bias in how per-task scores are aggregated into a final ranking.

Justifications

SL — The geometric convergence (d3) and geometry-probe interpretation (d2) establish the metric's validity; the aggregate bias (d1, OUT) is the aggregation-level artifact. Gating separates the *geometric truth* (cosine→Spearman measures a real semantic direction) from the *ranking artifact* (dataset-count weighting distorts the final score).

Antecedents (all must be IN):

  • IN evaluation-geometry-convergence — The convergence of embedding evaluation (cosine → Spearman correlation in SBERT/MTEB) and internal geometry analysis (covariance whitening in ROME, polytope geometry in Park) on the same second-moment structure reveals that standard benchmark evaluation is measuring the same geometric quantity that governs internal feature organization—evaluation and analysis are two readouts of one space.
  • IN embedding-evaluation-as-geometry-probe — The embedding evaluation pipeline (cosine similarity → Spearman correlation, validated by both SBERT and MTEB) is not an arbitrary similarity metric but a direct linear probe of the same universal feature geometry (polytopes, hierarchical orthogonality, sparse features) revealed by internal representation analysis; the observed task-specificity in MTEB scores reflects different task-specific linear projections of this shared geometric structure rather than a fundamental failure of the metric.

Unless (any of these IN defeats this justification):

  • IN mteb-aggregate-structural-bias — MTEB's aggregate "best model" ranking is structurally biased toward task-breadth coverage rather than semantic quality, because (a) the score is dataset-count-weighted (15 retrieval datasets dominate), (b) summarization is underrepresented by a single dataset, and (c) retrieval/clustering are English-only while other tasks are multilingual.