mteb-aggregate-structural-bias
IN derived (depth 1)
Created 2026-08-25T03:08:54+00:00 · Reviewed 2026-08-25T04:28:09+00:00
MTEB's aggregate "best model" ranking is structurally biased toward task-breadth coverage rather than semantic quality, because (a) the score is dataset-count-weighted (15 retrieval datasets dominate), (b) summarization is underrepresented by a single dataset, and (c) retrieval/clustering are English-only while other tasks are multilingual.
Summary
Picking the top-ranked model on MTEB mostly tells you which model handles the most English retrieval tasks, not which one genuinely understands language best, because the scoring gives retrieval roughly triple the weight of any other task, summarization is judged on a single dataset, and the largest task categories simply exclude non-English work. In practice, a model that performs adequately everywhere but excels at English retrieval will beat one with deeper multilingual or summarization skills, so the leaderboard can mislead anyone choosing a model for real, diverse, or non-English use.
Justifications
This belief has 3 justifications — it is IN if any one holds.
SL — Each bias independently distorts the aggregate ranking; the conclusion holds if any single structural asymmetry is present.
Antecedents (all must be IN):
- IN mteb-average-score-dataset-count-weighted — MTEB's overall average score is dataset-count-weighted rather than task-weighted, biasing results toward tasks with more datasets (retrieval, classification, clustering).
SL — Each bias independently distorts the aggregate ranking; the conclusion holds if any single structural asymmetry is present.
Antecedents (all must be IN):
- IN mteb-summarization-single-dataset — MTEB includes only one summarization dataset (SummEval, evaluated on CNN/DailyMail), making summarization the most imbalanced task in the benchmark.
SL — Each bias independently distorts the aggregate ranking; the conclusion holds if any single structural asymmetry is present.
Antecedents (all must be IN):
- IN mteb-retrieval-clustering-english-only — MTEB's retrieval and clustering tasks are English-only; multilingual coverage in MTEB exists only for classification, STS, and bitext mining tasks.
Dependents
These beliefs depend on this one:
- IN evaluation-as-readout-fidelity-measure — MTEB's dataset-count-weighted bias, SBERT's task-specificity gap, and the cosine→Spearman pipeline consistency are all manifestations of a single principle: embedding benchmarks measure readout fidelity (how faithfully a linear probe extracts a specific geometric direction) rather than the completeness or quality of the underlying representation space, making leaderboard rankings a statement about calibration rather than about semantics.
- OUT evaluation-geometry-valid-independent-of-aggregation — The convergence of the cosine→Spearman evaluation pipeline across SBERT and MTEB validates the *metric itself* as a geometric probe of semantic similarity, independent of any structural bias in how per-task scores are aggregated into a final ranking.
- OUT mteb-aggregate-as-readout-bias — MTEB's dataset-count-weighted aggregate score measures readout calibration breadth across task types rather than internal representation quality, because task-specificity is a readout-level phenomenon operating on top of a shared geometric substrate.
- OUT mteb-ranking-as-valid-comparator — The MTEB leaderboard provides a valid comparative quality signal for embedding models, with the cosine→Spearman pipeline serving as a geometry-consistent evaluation...