prh-clip-imagenet-finetuning-reduces-alignment
IN premise — summaries/2026/08/24/huh-2024-prh-s2-representations-are-converging-chunk-1.md
Created 2026-08-24T17:10:56+00:00
Fine-tuning CLIP on ImageNet classification reduces its representational alignment with LLMs, indicating that narrow-task specialization moves representations away from the platonic representation, while CLIP (with explicit language supervision) shows the highest vision-language alignment among tested models.
Summary
Specializing a vision model on a narrow task like ImageNet classification pulls its internal representations away from the shared space that connects images to language, effectively making it worse at aligning with how language models think. This matters because it shows a trade-off: the more you tune a model for one specific job, the further it drifts from the common representational ground that makes cross-modal understanding possible, which is why CLIP's original language-supervised training keeps it most aligned with text.