Abstract
We investigate ways to compose complex concepts in texts from primitive ones while grounding them in images. We propose Concept and Relation Graph (CRG), which builds on top of constituency analysis and consists of recursively combined concepts with predicate functions. Meanwhile, we propose a concept composition neural network called Composer to leverage the CRG for visually grounded concept learning. Specifically, we learn the grounding of both primitive and all composed concepts by aligning them to images and show that learning to compose leads to more robust grounding results, measured in text-to-image matching accuracy. Notably, our model can model grounded concepts forming at both the finer-grained sentence level and the coarser-grained intermediate level (or word-level). Composer leads to pronounced improvement in matching accuracy when the evaluation data has significant compound divergence from the training data.- Anthology ID:
- 2021.findings-emnlp.20
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2021
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- Findings
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 201–215
- Language:
- URL:
- https://aclanthology.org/2021.findings-emnlp.20
- DOI:
- 10.18653/v1/2021.findings-emnlp.20
- Cite (ACL):
- Bowen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, and Fei Sha. 2021. Visually Grounded Concept Composition. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 201–215, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Visually Grounded Concept Composition (Zhang et al., Findings 2021)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/2021.findings-emnlp.20.pdf
- Data
- Flickr30k