imagenet-decomposition-shows-coarse-but-not-fine-grained-alignment
IN premise — summaries/2026-08-24/koepke-2026-back-into-cave-s3-experimental-setup-chunk-1.md
Created 2026-08-24T17:11:00+00:00
On ImageNet at 49 images per class, DINOv2 retrieves correct-class neighbors 46.1% of the time and OpenLlama 58.0%, but strict cross-modal mutual kNN alignment (retrieving the same exact gallery item) remains flat at approximately 11%, indicating models share category-level structure but organize within-class structure differently.
Summary
Two different models (one for images, one for text) tend to agree on which broad category an item belongs to, but they organize the fine details within that category in idiosyncratic ways that don't line up. This means a system can't assume that cross-modal matching at the category level will transfer to matching at the instance level; the within-category geometry is model-specific, so any pipeline relying on exact cross-modal retrieval needs to account for that mismatch rather than treating it as noise.