Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case

Adam Dahlgren Lindström, Johanna Björklund, Suna Bensch, Frank Drewes

Abstract
Semantic embeddings have advanced the state of the art for countless natural language processing tasks, and various extensions to multimodal domains, such as visual-semantic embeddings, have been proposed. While the power of visual-semantic embeddings comes from the distillation and enrichment of information through machine learning, their inner workings are poorly understood and there is a shortage of analysis tools. To address this problem, we generalize the notion ofprobing tasks to the visual-semantic case. To this end, we (i) discuss the formalization of probing tasks for embeddings of image-caption pairs, (ii) define three concrete probing tasks within our general framework, (iii) train classifiers to probe for those properties, and (iv) compare various state-of-the-art embeddings under the lens of the proposed probing tasks. Our experiments reveal an up to 16% increase in accuracy on visual-semantic embeddings compared to the corresponding unimodal embeddings, which suggest that the text and image dimensions represented in the former do complement each other.
Anthology ID:
2020.coling-main.64
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Editors:
Donia Scott, Nuria Bel, Chengqing Zong
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
730–744
Language:
URL:
https://aclanthology.org/2020.coling-main.64
DOI:
10.18653/v1/2020.coling-main.64
Bibkey:
Cite (ACL):
Adam Dahlgren Lindström, Johanna Björklund, Suna Bensch, and Frank Drewes. 2020. Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case. In Proceedings of the 28th International Conference on Computational Linguistics, pages 730–744, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Probing Multimodal Embeddings for Linguistic Properties: the Visual-Semantic Case (Dahlgren Lindström et al., COLING 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2020.coling-main.64.pdf
Code
 dali-does/vse-probing
Data
MS COCO
PDF Search Code