ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation
Moran Yanuka, Morris Alper, Hadar Averbuch-Elor, Raja Giryes
Abstract
Web-scale training on paired text-image data is becoming increasingly central to multimodal learning, but is challenged by the highly noisy nature of datasets in the wild. Standard data filtering approaches succeed in removing mismatched text-image pairs, but permit semantically related but highly abstract or subjective text. These approaches lack the fine-grained ability to isolate the most concrete samples that provide the strongest signal for learning in a noisy dataset. In this work, we propose a new metric, Image Caption Concreteness (ICC), that evaluates caption text without an image reference to measure its concreteness and relevancy for use in multimodal learning. Our unsupervised approach leverages strong foundation models for measuring visual-semantic information loss in multimodal representations. We demonstrate that this strongly correlates with human evaluation of concreteness in both single-word and caption-level texts. Moreover, we show that curation using ICC complements existing approaches: It succeeds in selecting the highest quality samples from multimodal web-scale datasets to allow for efficient training in resource-constrained settings.- Anthology ID:
- 2024.findings-acl.657
- Volume:
- Findings of the Association for Computational Linguistics ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand and virtual meeting
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11048–11064
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.657
- DOI:
- Cite (ACL):
- Moran Yanuka, Morris Alper, Hadar Averbuch-Elor, and Raja Giryes. 2024. ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation. In Findings of the Association for Computational Linguistics ACL 2024, pages 11048–11064, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
- Cite (Informal):
- ICC : Quantifying Image Caption Concreteness for Multimodal Dataset Curation (Yanuka et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.657.pdf