Abstract
Image captioning models tend to describe images in an object-centric way, emphasising visible objects. But image descriptions can also abstract away from objects and describe the type of scene depicted. In this paper, we explore the potential of a state of the art Vision and Language model, VinVL, to caption images at the scene level using (1) a novel dataset which pairs images with both object-centric and scene descriptions. Through (2) an in-depth analysis of the effect of the fine-tuning, we show (3) that a small amount of curated data suffices to generate scene descriptions without losing the capability to identify object-level concepts in the scene; the model acquires a more holistic view of the image compared to when object-centric descriptions are generated. We discuss the parallels between these results and insights from computational and cognitive science research on scene perception.- Anthology ID:
- 2022.umios-1.6
- Volume:
- Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Wenjuan Han, Zilong Zheng, Zhouhan Lin, Lifeng Jin, Yikang Shen, Yoon Kim, Kewei Tu
- Venue:
- UM-IoS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 56–72
- Language:
- URL:
- https://aclanthology.org/2022.umios-1.6
- DOI:
- 10.18653/v1/2022.umios-1.6
- Cite (ACL):
- Michele Cafagna, Kees van Deemter, and Albert Gatt. 2022. Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions. In Proceedings of the Workshop on Unimodal and Multimodal Induction of Linguistic Structures (UM-IoS), pages 56–72, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions (Cafagna et al., UM-IoS 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.umios-1.6.pdf