task-specificity-vs-feature-universality
OUT derived (depth 1)
Created 2026-08-25T03:00:40+00:00 · Reviewed 2026-08-25T04:28:09+00:00
Embedding model quality is irreducibly task-specific (no single model dominates MTEB's 8 tasks) while internal feature representations are largely model-independent (SAE features transfer across architectures), separating output-space competition from internal geometric convergence.
Justifications
This belief has 2 justifications — it is IN if any one holds.
SL — MTEB demonstrates no universal champion across 8 task types and 33 models, yet SAEs produce similar features across GPT-2, Mistral, and Claude—revealing that internal representation geometry is far more universal than output-space performance.
Antecedents (all must be IN):
- IN mteb-no-dominant-embedding-model — Across the 33 models evaluated in MTEB, no single text embedding method dominates all tasks; different models top different task categories.
Unless (any of these IN defeats this justification):
- IN task-specificity-vs-feature-universality-v2 — Evaluation results suggest embedding model quality is substantially task-specific (no single model across 33 evaluated models dominates all MTEB task categories), while SAE-extracted features tend to be model-independent (more similar across architectures than to their own model's neurons), pointing toward a distinction between output-space performance differentiation and internal representational convergence.
SL — MTEB demonstrates no universal champion across 8 task types and 33 models, yet SAEs produce similar features across GPT-2, Mistral, and Claude—revealing that internal representation geometry is far more universal than output-space performance.
Antecedents (all must be IN):
- IN sae-universality-across-models — SAEs applied to different transformer models produce mostly similar features—more similar to each other than to their own model's neurons—suggesting features reflect data structure rather than architecture
Unless (any of these IN defeats this justification):
- IN task-specificity-vs-feature-universality-v2 — Evaluation results suggest embedding model quality is substantially task-specific (no single model across 33 evaluated models dominates all MTEB task categories), while SAE-extracted features tend to be model-independent (more similar across architectures than to their own model's neurons), pointing toward a distinction between output-space performance differentiation and internal representational convergence.
Dependents
These beliefs depend on this one:
- IN embedding-evaluation-as-geometry-probe — The embedding evaluation pipeline (cosine similarity → Spearman correlation, validated by both SBERT and MTEB) is not an arbitrary similarity metric but a direct linear probe of the same universal feature geometry (polytopes, hierarchical orthogonality, sparse features) revealed by internal representation analysis; the observed task-specificity in MTEB scores reflects different task-specific linear projections of this shared geometric structure rather than a fundamental failure of the metric.
- OUT mteb-no-dominant-model-as-geometric-signature — MTEB's observation that no single embedding model dominates all eight tasks is the expected geometric signature of a shared canonical semantic space probed by task-specific linear readouts, and the same second-moment structure that generates this task-specificity pattern is precisely what makes rank-one knowledge editing possible.
- OUT task-specificity-emerges-from-readout — Task-specificity in embedding quality is a readout phenomenon: the internal feature geometry is largely model-independent (convergent across architectures), while MTEB's no-dominant-model result arises because each task's unembedding/projection head selects a different subspace of the same shared geometric structure.