Abstract
Despite the impressive advancements achieved through vision-and-language pretraining, it remains unclear whether multi-modal learning can help understand each individual modality. In this work, we conduct a comparative analysis of the visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks. Five probing tasks are evaluated in order to assess the quality of the learned representations in a nuanced manner. Our results on five probing tasks suggest vision-and-language models are better at label prediction tasks like object and attribute prediction, while vision-only models are stronger at dense prediction tasks that require more localized information. We hope our study sheds light on the role of language in visual learning, and serves as an empirical guide for various pretrained models.- Anthology ID:
- 2024.eacl-long.146
- Volume:
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Yvette Graham, Matthew Purver
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2378–2390
- Language:
- URL:
- https://aclanthology.org/2024.eacl-long.146
- DOI:
- Cite (ACL):
- Zhuowan Li, Cihang Xie, Benjamin Van Durme, and Alan Yuille. 2024. Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2378–2390, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models (Li et al., EACL 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.eacl-long.146.pdf