ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation

Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, Tristan Thrush

Abstract
This paper introduces the ColorSwap dataset, designed to assess and improve the proficiency of multimodal models in matching objects with their colors. The dataset is comprised of 2,000 unique image-caption pairs, grouped into 1,000 examples. Each example includes a caption-image pair, along with a “color-swapped” pair. We follow the Winoground schema: the two captions in an example have the same words, but the color words have been rearranged to modify different objects. The dataset was created through a novel blend of automated caption and image generation with humans in the loop. We evaluate image-text matching (ITM) and visual language models (VLMs) and find that even the latest ones are still not robust at this task. GPT-4V and LLaVA score 72% and 42% on our main VLM metric, although they may improve with more advanced prompting techniques. On the main ITM metric, contrastive models such as CLIP and SigLIP perform close to chance (at 12% and 30%, respectively), although the non-contrastive BLIP ITM model is stronger (87%). We also find that finetuning on fewer than 2,000 examples yields significant performance gains on this out-of-distribution word-order understanding task.
Anthology ID:
2024.findings-acl.99
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1716–1726
Language:
URL:
https://aclanthology.org/2024.findings-acl.99
DOI:
10.18653/v1/2024.findings-acl.99
Bibkey:
Cite (ACL):
Jirayu Burapacheep, Ishan Gaur, Agam Bhatia, and Tristan Thrush. 2024. ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1716–1726, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
ColorSwap: A Color and Word Order Dataset for Multimodal Evaluation (Burapacheep et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/autopr/2024.findings-acl.99.pdf
PDF Search Fix metadata