evaluation-as-readout-fidelity-measure
IN derived (depth 3)
Created 2026-08-25T03:11:44+00:00 · Reviewed 2026-08-25T04:28:09+00:00
MTEB's dataset-count-weighted bias, SBERT's task-specificity gap, and the cosine→Spearman pipeline consistency are all manifestations of a single principle: embedding benchmarks measure readout fidelity (how faithfully a linear probe extracts a specific geometric direction) rather than the completeness or quality of the underlying representation space, making leaderboard rankings a statement about calibration rather than about semantics.
Summary
What embedding leaderboards actually measure is how cleanly a simple linear tool can extract one specific direction from the model's geometric layout, not how rich or complete the model's internal understanding is. A top ranking therefore says the model's space is well-calibrated for extraction, not that it truly grasps language better than a lower-ranked one, so treating those numbers as a semantic quality score is a category error.
Justifications
This belief has 3 justifications — it is IN if any one holds.
SL — Each antecedent independently establishes that evaluation measures the readout/probe rather than the representation itself (MTEB's structural bias, SBERT's task-specificity, and the geometry-probe interpretation); the unified "readout fidelity" claim is supported by convergent evidence from any one.
Antecedents (all must be IN):
- IN mteb-aggregate-structural-bias — MTEB's aggregate "best model" ranking is structurally biased toward task-breadth coverage rather than semantic quality, because (a) the score is dataset-count-weighted (15 retrieval datasets dominate), (b) summarization is underrepresented by a single dataset, and (c) retrieval/clustering are English-only while other tasks are multilingual.
SL — Each antecedent independently establishes that evaluation measures the readout/probe rather than the representation itself (MTEB's structural bias, SBERT's task-specificity, and the geometry-probe interpretation); the unified "readout fidelity" claim is supported by convergent evidence from any one.
Antecedents (all must be IN):
- OUT task-specificity-emerges-from-readout — Task-specificity in embedding quality is a readout phenomenon: the internal feature geometry is largely model-independent (convergent across architectures), while MTEB's no-dominant-model result arises because each task's unembedding/projection head selects a different subspace of the same shared geometric structure.
SL — Each antecedent independently establishes that evaluation measures the readout/probe rather than the representation itself (MTEB's structural bias, SBERT's task-specificity, and the geometry-probe interpretation); the unified "readout fidelity" claim is supported by convergent evidence from any one.
Antecedents (all must be IN):
- IN embedding-evaluation-as-geometry-probe — The embedding evaluation pipeline (cosine similarity → Spearman correlation, validated by both SBERT and MTEB) is not an arbitrary similarity metric but a direct linear probe of the same universal feature geometry (polytopes, hierarchical orthogonality, sparse features) revealed by internal representation analysis; the observed task-specificity in MTEB scores reflects different task-specific linear projections of this shared geometric structure rather than a fundamental failure of the metric.
Dependents
These beliefs depend on this one:
- OUT mteb-task-diversity-as-geometric-anisotropy — MTEB's "no dominant model" result across 8 task types is the expected geometric signature of anisotropic readout in a shared semantic space: each task type (STS, retrieval, clustering, classification) probes a different projection direction, and task-specificity is the *predicted* outcome of geometric anisotropy, not a benchmark failure or model deficiency.