@inproceedings{liu-etal-2025-data-language,
    title = "Data or Language Supervision: What Makes {CLIP} Better than {DINO}?",
    author = "Liu, Yiming  and
      Zhang, Yuhui  and
      Ghosh, Dhruba  and
      Schmidt, Ludwig  and
      Yeung-Levy, Serena",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.98/",
    doi = "10.18653/v1/2025.findings-emnlp.98",
    pages = "1868--1874",
    ISBN = "979-8-89176-335-7",
    abstract = "CLIP outperforms self-supervised models like DINO as vision encoders for vision-language models (VLMs), but it remains unclear whether this advantage stems from CLIP{'}s language supervision or its much larger training data. To disentangle these factors, we pre-train CLIP and DINO under controlled settings{---}using the same architecture, dataset, and training configuration{---}achieving similar ImageNet accuracy. Embedding analysis shows that CLIP captures high-level semantics (e.g., object categories, text), while DINO is more responsive to low-level features like colors and styles. When integrated into VLMs and evaluated on 20 VQA benchmarks, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones. Variants of language supervision (e.g., sigmoid loss, pre-trained language encoders) yield limited gains. Our findings provide scientific insights into vision encoder design and its impact on VLM performance."
}