Abstract
Scene graphs represent semantic information in images, which can help image captioning system to produce more descriptive outputs versus using only the image as context. Recent captioning approaches rely on ad-hoc approaches to obtain graphs for images. However, those graphs introduce noise and it is unclear the effect of parser errors on captioning accuracy. In this work, we investigate to what extent scene graphs can help image captioning. Our results show that a state-of-the-art scene graph parser can boost performance almost as much as the ground truth graphs, showing that the bottleneck currently resides more on the captioning models than on the performance of the scene graph parser.- Anthology ID:
- D19-6405
- Volume:
- Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN)
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Editors:
- Aditya Mogadala, Dietrich Klakow, Sandro Pezzelle, Marie-Francine Moens
- Venue:
- WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 29–34
- Language:
- URL:
- https://aclanthology.org/D19-6405
- DOI:
- 10.18653/v1/D19-6405
- Cite (ACL):
- Dalin Wang, Daniel Beck, and Trevor Cohn. 2019. On the Role of Scene Graphs in Image Captioning. In Proceedings of the Beyond Vision and LANguage: inTEgrating Real-world kNowledge (LANTERN), pages 29–34, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- On the Role of Scene Graphs in Image Captioning (Wang et al., 2019)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/D19-6405.pdf