Abstract
Standard image caption generation systems produce generic descriptions of images and do not utilize any contextual information or world knowledge. In particular, they are unable to generate captions that contain references to the geographic context of an image, for example, the location where a photograph is taken or relevant geographic objects around an image location. In this paper, we develop a geo-aware image caption generation system, which incorporates geographic contextual information into a standard image captioning pipeline. We propose a way to build an image-specific representation of the geographic context and adapt the caption generation network to produce appropriate geographic names in the image descriptions. We evaluate our system on a novel captioning dataset that contains contextualized captions and geographic metadata and achieve substantial improvements in BLEU, ROUGE, METEOR and CIDEr scores. We also introduce a new metric to assess generated geographic references directly and empirically demonstrate our system’s ability to produce captions with relevant and factually accurate geographic referencing.- Anthology ID:
- 2020.coling-main.280
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Editors:
- Donia Scott, Nuria Bel, Chengqing Zong
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 3143–3156
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.280
- DOI:
- 10.18653/v1/2020.coling-main.280
- Cite (ACL):
- Sofia Nikiforova, Tejaswini Deoskar, Denis Paperno, and Yoad Winter. 2020. Geo-Aware Image Caption Generation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3143–3156, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- Geo-Aware Image Caption Generation (Nikiforova et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2020.coling-main.280.pdf
- Data
- MS COCO