mteb-no-code-or-multimodal-datasets

IN premise — summaries/2026/08/24/muennighoff-2022-mteb-s3-multinguality-mteb-contains-multilingual.md

Created 2026-08-25T02:58:20+00:00

MTEB includes no code datasets and no cross-modal (image/text) evaluation; CodeSearchNet, TyDi QA, XOR QA, and MIRACL are suggested as future extensions.

Summary

MTEB is a text-only benchmark for comparing embedding models, so it says nothing about how those models perform on code search or image-text matching tasks. Anyone relying on MTEB scores to pick a model is getting an incomplete picture that excludes two major use cases.