mteb-ranking-as-valid-comparator
OUT derived (depth 3)
Created 2026-08-25T03:10:25+00:00
The MTEB leaderboard provides a valid comparative quality signal for embedding models, with the cosine→Spearman pipeline serving as a geometry-consistent evaluation...
Justifications
SL — Validity requires both a consistent pipeline AND meaningful cross-task comparison (readout-level specificity). The structural bias (dataset-count weighting, single-dataset summarization) invalidates the aggregate as a quality comparator, making this claim OUT while the bias is IN.
Antecedents (all must be IN):
- IN embedding-evaluation-pipeline-consistency — The SBERT pipeline (NLI fine-tuning → mean pooling → cosine similarity → Spearman correlation) is the de-facto standard validated across both the original SBERT paper and the MTEB benchmark's STS evaluation protocol.
- OUT task-specificity-emerges-from-readout — Task-specificity in embedding quality is a readout phenomenon: the internal feature geometry is largely model-independent (convergent across architectures), while MTEB's no-dominant-model result arises because each task's unembedding/projection head selects a different subspace of the same shared geometric structure.
Unless (any of these IN defeats this justification):
- IN mteb-aggregate-structural-bias — MTEB's aggregate "best model" ranking is structurally biased toward task-breadth coverage rather than semantic quality, because (a) the score is dataset-count-weighted (15 retrieval datasets dominate), (b) summarization is underrepresented by a single dataset, and (c) retrieval/clustering are English-only while other tasks are multilingual.