Visually Grounded Concept Composition

Bowen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, Fei Sha


Abstract
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images. We propose Concept and Relation Graph (CRG), which builds on top of constituency analysis and consists of recursively combined concepts with predicate functions. Meanwhile, we propose a concept composition neural network called Composer to leverage the CRG for visually grounded concept learning. Specifically, we learn the grounding of both primitive and all composed concepts by aligning them to images and show that learning to compose leads to more robust grounding results, measured in text-to-image matching accuracy. Notably, our model can model grounded concepts forming at both the finer-grained sentence level and the coarser-grained intermediate level (or word-level). Composer leads to pronounced improvement in matching accuracy when the evaluation data has significant compound divergence from the training data.
Anthology ID:
2021.findings-emnlp.20
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
201–215
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.20
DOI:
10.18653/v1/2021.findings-emnlp.20
Bibkey:
Cite (ACL):
Bowen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, and Fei Sha. 2021. Visually Grounded Concept Composition. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 201–215, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Visually Grounded Concept Composition (Zhang et al., Findings 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2021.findings-emnlp.20.pdf
Video:
 https://preview.aclanthology.org/auto-file-uploads/2021.findings-emnlp.20.mp4
Data
Flickr30k