Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements

Andrey Savchenko, Anton Alekseev, Sejeong Kwon, Elena Tutubalina, Evgeny Myasnikov, Sergey Nikolenko

Abstract
Understanding image advertisements is a challenging task, often requiring non-literal interpretation. We argue that standard image-based predictions are insufficient for symbolism prediction. Following the intuition that texts and images are complementary in advertising, we introduce a multimodal ensemble of a state of the art image-based classifier, a classifier based on an object detection architecture, and a fine-tuned language model applied to texts extracted from ads by OCR. The resulting system establishes a new state of the art in symbolism prediction.
Anthology ID:
2020.coling-main.171
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
1886–1892
Language:
URL:
https://aclanthology.org/2020.coling-main.171
DOI:
10.18653/v1/2020.coling-main.171
Bibkey:
Cite (ACL):
Andrey Savchenko, Anton Alekseev, Sejeong Kwon, Elena Tutubalina, Evgeny Myasnikov, and Sergey Nikolenko. 2020. Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1886–1892, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Ad Lingua: Text Classification Improves Symbolism Prediction in Image Advertisements (Savchenko et al., COLING 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2020.coling-main.171.pdf
Data
Open Images V4
PDF Search