mteb-add-dataset-single-config-file

IN premise — summaries/2026/08/24/muennighoff-2022-mteb-s1-introduction.md

Created 2026-08-25T02:58:18+00:00

Adding a new dataset for an existing MTEB task requires a single configuration file specifying the task name and a Hugging Face dataset identifier; a new task requires implementing a task interface plus an evaluator class.

Summary

In MTEB, plugging in a new test dataset for a task the benchmark already supports is a lightweight job—just one small config file naming the task and pointing to a hosted dataset. Inventing a brand-new task type, by contrast, is a much heavier engineering lift that requires writing custom code for both the task definition and the scoring logic, so the two paths carry very different cost and risk.