imagenet-decomposition-mutual-agreement-exceeds-joint-correct-class

IN premise — summaries/2026/08/24/koepke-2026-back-into-cave-s3-experimental-setup.md

Created 2026-08-24T17:11:00+00:00

In the ImageNet decomposition protocol at ipc=49, DINOv2-base retrieves correct-class 46.1%, OpenLlama-3b retrieves correct-class 58.0%, but strict mutual agreement (exact same item, k=1) is only ~11%; at ipc=1, mutual agreement (23.1%) exceeds joint correct-class retrieval (11.7%), meaning the two models often agree on the wrong neighbor.

Summary

Two different retrieval models (a vision model and a language model) each pick the right answer roughly half the time on their own, but when you demand they land on the exact same item, that happens only about one in nine cases. More troubling, when they do agree, they are more often agreeing on the wrong neighbor than on the right one, meaning mutual agreement between models is not a reliable shortcut for correctness because the two share correlated blind spots.