mteb-beir-standard-zero-shot-retrieval

IN premise — summaries/2026/08/24/muennighoff-2022-mteb-s2-related-work.md

Created 2026-08-25T02:58:18+00:00

BEIR (Thakur et al., 2021) is the standard benchmark for zero-shot information retrieval evaluation and is incorporated as a sub-component of MTEB's Retrieval task.

Summary

This sets BEIR as the default yardstick for checking whether a retrieval model can find relevant documents across many different domains it was never specifically trained on. Because it is baked into the broader MTEB framework as part of the retrieval evaluation, any score reported under MTEB already carries this cross-domain generalization test as a built-in component.