mteb-sts-main-metric
IN premise — summaries/2026/08/24/muennighoff-2022-mteb-s8-tasks.md
Created 2026-08-25T02:58:20+00:00
The main metric for MTEB STS and Summarization evaluation is Spearman correlation based on cosine similarity
Summary
For STS and Summarization tasks in the MTEB benchmark, the system is judged on whether its similarity scores rank sentence pairs in the correct order, not on matching exact score values. This means the benchmark rewards getting the relative ordering of similarities right, so a model that slightly underestimates all scores but preserves the ranking will score well, while one that scrambles the ordering will score poorly.
Dependents
These beliefs depend on this one:
- IN embedding-evaluation-pipeline-consistency — The SBERT pipeline (NLI fine-tuning → mean pooling → cosine similarity → Spearman correlation) is the de-facto standard validated across both the original SBERT paper and the MTEB benchmark's STS evaluation protocol.