Embedding Benchmarks

54 beliefs (54 IN, 0 OUT)

This topic centers on the Massive Text Embedding Benchmark (MTEB) and its role as a unified evaluation framework for text embedding models. MTEB, introduced by Muennighoff et al. (2022, arXiv 2210.07316v3) by researchers at Hugging Face and Cohere, covers 8 task types, 58 constituent datasets, 10 multilingual datasets, and 112 languages (mteb-scope-8-task-types-58-datasets-112-languages, mteb-112-languages). It fills a gap left by prior benchmarks such as SemEval/SentEval (STS and classifier evaluation only), USEB (reranking), and BEIR (zero-shot retrieval), none of which offered a single cross-task comparison (mteb-prior-benchmarks-narrow-scope). The benchmark's scope is bounded: it excludes code datasets and cross-modal evaluation, with CodeSearchNet, TyDi QA, XOR QA, and MIRACL noted as future extensions (mteb-no-code-or-multimodal-datasets). Across approximately 5,000 experiments on 30+ models, MTEB found that no single embedding method dominates all task categories (mteb-no-dominant-embedding-model), a finding consistent with the benchmark's broader interpretive claim.

The most consequential belief in this group is the interpretive principle that embedding benchmarks measure readout fidelity rather than representational completeness (evaluation-as-readout-fidelity-measure). Under this reading, the specific evaluation pipelines are telling: classification uses a logistic regression head with max_iter=100 trained on frozen embeddings (mteb-classification-head-params, mteb-classification-logistic-regression-100-iterations), clustering uses mini-batch k-means with batch size 32 scored by V-measure (mteb-clustering-params, mteb-clustering-vmeasure-kmeans-batch32), STS and summarization rely on Spearman correlation of cosine similarities (mteb-sts-main-metric, mteb-sts-evaluation-metric), bitext mining uses F1 over closest cosine pairs (mteb-bitext-mining-metric-f1), pair classification reports average precision via cosine (mteb-pair-class-main-metric), and reranking uses MAP (mteb-reranking-main-metric). All of these are linear or near-linear probes that extract a specific geometric direction from the embedding space. Combined with MTEB's dataset-count-weighted scoring (which biases toward retrieval, classification, and clustering simply because they have more datasets, mteb-average-score-dataset-count-weighted) and the cosine-to-Spearman pipeline, the leaderboard becomes a statement about calibration fidelity rather than about the semantic richness of the underlying space. The GloVe baseline, which applies a context-agnostic word-to-vector averaging layer and scores roughly 42 on average (mteb-glove-baseline-avg-score, mteb-glove-baseline-average-42, mteb-glove-averaging-layer), establishes a floor for learned embeddings, while ST5-XXL leads English scores at 59.51, followed by GTR-XXL (58.97) and SGPT-5.8B-msmarco (58.81) (st5-xxl-best-avg-score). In multilingual settings, LaBSE tops bitext mining (labse-top-bitext-mining) and MPNet leads classification and STS (mpnet-best-multilingual-cls-sts), reinforcing the no-dominant-model pattern.

The remaining beliefs in this group are largely premise-level: they fix concrete parameters, dataset characteristics, and practical infrastructure. These include the specific language counts per multilingual dataset (Tatoeba=112, Massive=51, STS22=18, BUCC=5, mteb-multilingual-dataset-language-counts, mteb-bucc-5-languages, mteb-massive-51-languages, sts22-language-pairs), dataset sizes (BIOSS/BIOSSES at exactly 100 pairs, SICK-R at roughly 10,000, STSBenchmark as monolingual English, mteb-bioss-100-sentence-pairs, mteb-biosses-100-biomedical-pairs, mteb-sickr-approximately-10000-pairs, stsbenchmark-monolingual-english), the input-type taxonomy of s2s, p2p, and s2p (mteb-input-type-taxonomy), embedding sizes ranging from 1.2 kB (GloVe) to 16.4 kB (SGPT-5.8B) (mteb-embedding-size-range), text lengths from ~32 to ~1981 characters (mteb-text-length-range), the summarization task's single-dataset imbalance (mteb-summarization-single-dataset), clustering and reranking's strong inter-task correlation versus summarization's outlier status (mteb-clustering-reranking-strongest-correlation), and the arXiv clustering's three split strategies (mteb-arxiv-clustering-three-splits). Practical details cover the code repository at github.com/embeddings-benchmark/mteb (mteb-code-repo, mteb-github-repository), the Hugging Face leaderboard and JSON results (mteb-leaderboard-url, mteb-results-hosted-huggingface, mteb-results-json-url), the ease of adding models (fewer than 10 lines) or datasets (a single config file) (mteb-add-model-10-lines-code, mteb-add-dataset-single-config-file), speed benchmarking on a single A100 80GB (mteb-speed-bench-hardware), and the Jean Zay cluster at IDRIS used for evaluation runs (mteb-hpc-jean-zay). STS scores range from 0 to 5 (mteb-sts-score-range-0-to-5), and 56 of the 58 datasets are English subsets in the main results table (mteb-total-datasets-count).

All beliefs in this topic are marked IN; none have been retracted. The interpretive belief about readout fidelity (evaluation-as-readout-fidelity-measure) functions as a derived or synthetic claim that draws on the methodological premises (linear probes, dataset-count weighting, cosine-based metrics) to reframe what the leaderboard numbers actually signify. The model-performance beliefs (GloVe baseline, ST5-XXL top score, LaBSE and MPNet multilingual leadership) are empirical observations that are consistent with the readout-fidelity reading, since different tasks probe different geometric directions and no single model excels at all of them simultaneously.