cross-modal-mutual-knn-drops-with-gallery-scale

IN premise — summaries/2026/08/24/koepke-2026-back-into-cave-s3-experimental-setup.md

Created 2026-08-24T17:11:00+00:00

Mutual kNN alignment between DINOv2-base and OpenLlama-3b drops from 0.135 (k=10) / 0.058 (k=1) on WIT-1024 to 0.008 (k=10) / 0.001 (k=1) on LAION-15M, while within-modality mutual kNN (OpenLlama-3b vs 13b at k=1) stays in [0.59, 0.62] and DINOv2-base vs giant in [0.35, 0.45] across all gallery sizes.

Summary

Cross-modal agreement between the vision encoder and the language model essentially vanishes once the comparison pool grows from ~1,000 to ~15 million items, dropping by roughly a factor of ten to a hundred, while same-modality model pairs (e.g., Llama-3b vs 13b, DINOv2-base vs giant) keep their alignment stable across all pool sizes. This matters because it means small-gallery benchmarks dramatically overstate how well vision and language representations actually line up at scale, so any system that relies on cross-modal retrieval or alignment should not trust patterns seen on tiny test sets.