Abstract
The challenge of automatically describing images and videos has stimulated much research in Computer Vision and Natural Language Processing. In order to test the semantic abilities of new algorithms, we need reliable and objective ways of measuring progress. We show that standard evaluation measures do not take into account the semantic richness of a description, and give the impression that sparse machine descriptions outperform rich human descriptions. We introduce and test a new measure of semantic ability based on relative lexical diversity. We show how our measure can work alongside existing measures to achieve state of the art correlation with human judgement of quality. We also introduce a new dataset: Rich-Sparse Descriptions, which provides 2K human and machine descriptions to stimulate interest into the semantic evaluation of machine descriptions.- Anthology ID:
- W19-0603
- Volume:
- Proceedings of the 13th International Conference on Computational Semantics - Student Papers
- Month:
- May
- Year:
- 2019
- Address:
- Gothenburg, Sweden
- Editors:
- Simon Dobnik, Stergios Chatzikyriakidis, Vera Demberg, Kathrein Abu Kwaik, Vladislav Maraev
- Venue:
- IWCS
- SIG:
- SIGSEM
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15–23
- Language:
- URL:
- https://aclanthology.org/W19-0603
- DOI:
- 10.18653/v1/W19-0603
- Cite (ACL):
- Austin Kershaw and Miroslaw Bober. 2019. The Lexical Gap: An Improved Measure of Automated Image Description Quality. In Proceedings of the 13th International Conference on Computational Semantics - Student Papers, pages 15–23, Gothenburg, Sweden. Association for Computational Linguistics.
- Cite (Informal):
- The Lexical Gap: An Improved Measure of Automated Image Description Quality (Kershaw & Bober, IWCS 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/W19-0603.pdf