Abstract
We study verbs in image–text corpora, contrasting caption corpora, where texts are explicitly written to characterize image content, with depiction corpora, where texts and images may stand in more general relations. Captions show a distinctively limited distribution of verbs, with strong preferences for specific tense, aspect, lexical aspect, and semantic field. These limitations, which appear in data elicited by a range of methods, restrict the utility of caption corpora to inform image retrieval, multimodal document generation, and perceptually-grounded semantic models. We suggest that these limitations reflect the discourse constraints in play when subjects write texts to accompany imagery, so we argue that future development of image–text corpora should work to increase the diversity of event descriptions, while looking explicitly at the different ways text and imagery can be coherently related.- Anthology ID:
- W19-1806
- Volume:
- Proceedings of the Second Workshop on Shortcomings in Vision and Language
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Raffaella Bernardi, Raquel Fernandez, Spandana Gella, Kushal Kafle, Christopher Kanan, Stefan Lee, Moin Nabi
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 58–67
- Language:
- URL:
- https://aclanthology.org/W19-1806
- DOI:
- 10.18653/v1/W19-1806
- Cite (ACL):
- Malihe Alikhani and Matthew Stone. 2019. “Caption” as a Coherence Relation: Evidence and Implications. In Proceedings of the Second Workshop on Shortcomings in Vision and Language, pages 58–67, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- “Caption” as a Coherence Relation: Evidence and Implications (Alikhani & Stone, NAACL 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/W19-1806.pdf
- Data
- Flickr30k, MS COCO, VIST