Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding
Abhidip Bhattacharyya, Cecilia Mauceri, Martha Palmer, Christoffer Heckman
Abstract
As vision processing and natural language processing continue to advance, there is increasing interest in multimodal applications, such as image retrieval, caption generation, and human-robot interaction. These tasks require close alignment between the information in the images and text. In this paper, we present a new multimodal dataset that combines state of the art semantic annotation for language with the bounding boxes of corresponding images. This richer multimodal labeling supports cross-modal inference for applications in which such alignment is useful. Our semantic representations, developed in the natural language processing community, abstract away from the surface structure of the sentence, focusing on specific actions and the roles of their participants, a level that is equally relevant to images. We then utilize these representations in the form of semantic role labels in the captions and the images and demonstrate improvements in standard tasks such as image retrieval. The potential contributions of these additional labels is evaluated using a role-aware retrieval system based on graph convolutional and recurrent neural networks. The addition of semantic roles into this system provides a significant increase in capability and greater flexibility for these tasks, and could be extended to state-of-the-art techniques relying on transformers with larger amounts of annotated data.- Anthology ID:
- 2022.lrec-1.528
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 4944–4954
- Language:
- URL:
- https://aclanthology.org/2022.lrec-1.528
- DOI:
- Cite (ACL):
- Abhidip Bhattacharyya, Cecilia Mauceri, Martha Palmer, and Christoffer Heckman. 2022. Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4944–4954, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Aligning Images and Text with Semantic Role Labels for Fine-Grained Cross-Modal Understanding (Bhattacharyya et al., LREC 2022)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/2022.lrec-1.528.pdf