mutual-knn-scores-drop-sharply-with-gallery-scale
IN premise — summaries/2026-08-24/koepke-2026-back-into-cave-s3-experimental-setup-chunk-1.md
Created 2026-08-24T17:11:00+00:00
Cross-modal mutual kNN alignment scores between DINOv2-base and OpenLlama-3b fall from 0.135 (k=10) and 0.058 (k=1) on WIT-1024 to 0.008 (k=10) and 0.001 (k=1) on LAION-15M, demonstrating a sharp degradation of measured alignment as gallery size increases by ~15,000x.
Summary
When you only compare a small set of images and text, the vision and language models look moderately aligned, but that alignment almost completely vanishes once the pool of candidates grows to real-world scale (around 15,000x larger). In practical terms, any system built on these two models finding each other in a shared embedding space will not hold up beyond toy-sized datasets, because the cross-modal matching signal is so weak that it gets drowned out by the sheer number of near-miss candidates at scale.