Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction

Yue Yang, Artemis Panagopoulou, Marianna Apidianaki, Mark Yatskar, Chris Callison-Burch


Abstract
Neural language models encode rich knowledge about entities and their relationships which can be extracted from their representations using probing. Common properties of nouns (e.g., red strawberries, small ant) are, however, more challenging to extract compared to other types of knowledge because they are rarely explicitly stated in texts.We hypothesize this to mainly be the case for perceptual properties which are obvious to the participants in the communication. We propose to extract these properties from images and use them in an ensemble model, in order to complement the information that is extracted from language models. We consider perceptual properties to be more concrete than abstract properties (e.g., interesting, flawless). We propose to use the adjectives’ concreteness score as a lever to calibrate the contribution of each source (text vs. images). We evaluate our ensemble model in a ranking task where the actual properties of a noun need to be ranked higher than other non-relevant properties. Our results show that the proposed combination of text and images greatly improves noun property prediction compared to powerful text-based language models.
Anthology ID:
2022.findings-emnlp.45
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2022
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
638–655
Language:
URL:
https://aclanthology.org/2022.findings-emnlp.45
DOI:
Bibkey:
Cite (ACL):
Yue Yang, Artemis Panagopoulou, Marianna Apidianaki, Mark Yatskar, and Chris Callison-Burch. 2022. Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 638–655, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction (Yang et al., Findings 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.findings-emnlp.45.pdf