Learning Representations Specialized in Spatial Knowledge: Leveraging Language and Vision

Guillem Collell, Marie-Francine Moens


Abstract
Spatial understanding is crucial in many real-world problems, yet little progress has been made towards building representations that capture spatial knowledge. Here, we move one step forward in this direction and learn such representations by leveraging a task consisting in predicting continuous 2D spatial arrangements of objects given object-relationship-object instances (e.g., “cat under chair”) and a simple neural network model that learns the task from annotated images. We show that the model succeeds in this task and, furthermore, that it is capable of predicting correct spatial arrangements for unseen objects if either CNN features or word embeddings of the objects are provided. The differences between visual and linguistic features are discussed. Next, to evaluate the spatial representations learned in the previous task, we introduce a task and a dataset consisting in a set of crowdsourced human ratings of spatial similarity for object pairs. We find that both CNN (convolutional neural network) features and word embeddings predict human judgments of similarity well and that these vectors can be further specialized in spatial knowledge if we update them when training the model that predicts spatial arrangements of objects. Overall, this paper paves the way towards building distributed spatial representations, contributing to the understanding of spatial expressions in language.
Anthology ID:
Q18-1010
Volume:
Transactions of the Association for Computational Linguistics, Volume 6
Month:
Year:
2018
Address:
Cambridge, MA
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
133–144
Language:
URL:
https://aclanthology.org/Q18-1010
DOI:
10.1162/tacl_a_00010
Bibkey:
Cite (ACL):
Guillem Collell and Marie-Francine Moens. 2018. Learning Representations Specialized in Spatial Knowledge: Leveraging Language and Vision. Transactions of the Association for Computational Linguistics, 6:133–144.
Cite (Informal):
Learning Representations Specialized in Spatial Knowledge: Leveraging Language and Vision (Collell & Moens, TACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/Q18-1010.pdf
Code
 gcollell/spatial-representations
Data
Visual Genome