Object Counts! Bringing Explicit Detections Back into Image Captioning

Josiah Wang, Pranava Swaroop Madhyastha, Lucia Specia

Abstract
The use of explicit object detectors as an intermediate step to image captioning – which used to constitute an essential stage in early work – is often bypassed in the currently dominant end-to-end approaches, where the language model is conditioned directly on a mid-level image embedding. We argue that explicit detections provide rich semantic information, and can thus be used as an interpretable representation to better understand why end-to-end image captioning systems work well. We provide an in-depth analysis of end-to-end image captioning by exploring a variety of cues that can be derived from such object detections. Our study reveals that end-to-end image captioning systems rely on matching image representations to generate captions, and that encoding the frequency, size and position of objects are complementary and all play a role in forming a good image representation. It also reveals that different object categories contribute in different ways towards image captioning.
Anthology ID:
N18-1198
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2180–2193
Language:
URL:
https://aclanthology.org/N18-1198
DOI:
10.18653/v1/N18-1198
Bibkey:
Cite (ACL):
Josiah Wang, Pranava Swaroop Madhyastha, and Lucia Specia. 2018. Object Counts! Bringing Explicit Detections Back into Image Captioning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2180–2193, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Object Counts! Bringing Explicit Detections Back into Image Captioning (Wang et al., NAACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/proper-vol2-ingestion/N18-1198.pdf
Data
MS COCO
PDF Search