Guiding Interaction Behaviors for Multi-modal Grounded Language Learning

Jesse Thomason, Jivko Sinapov, Raymond Mooney

Abstract
Multi-modal grounded language learning connects language predicates to physical properties of objects in the world. Sensing with multiple modalities, such as audio, haptics, and visual colors and shapes while performing interaction behaviors like lifting, dropping, and looking on objects enables a robot to ground non-visual predicates like “empty” as well as visual predicates like “red”. Previous work has established that grounding in multi-modal space improves performance on object retrieval from human descriptions. In this work, we gather behavior annotations from humans and demonstrate that these improve language grounding performance by allowing a system to focus on relevant behaviors for words like “white” or “half-full” that can be understood by looking or lifting, respectively. We also explore adding modality annotations (whether to focus on audio or haptics when performing a behavior), which improves performance, and sharing information between linguistically related predicates (if “green” is a color, “white” is a color), which improves grounding recall but at the cost of precision.
Anthology ID:
W17-2803
Volume:
Proceedings of the First Workshop on Language Grounding for Robotics
Month:
August
Year:
2017
Address:
Vancouver, Canada
Editors:
Mohit Bansal, Cynthia Matuszek, Jacob Andreas, Yoav Artzi, Yonatan Bisk
Venue:
RoboNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
20–24
Language:
URL:
https://aclanthology.org/W17-2803
DOI:
10.18653/v1/W17-2803
Bibkey:
Cite (ACL):
Jesse Thomason, Jivko Sinapov, and Raymond Mooney. 2017. Guiding Interaction Behaviors for Multi-modal Grounded Language Learning. In Proceedings of the First Workshop on Language Grounding for Robotics, pages 20–24, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Guiding Interaction Behaviors for Multi-modal Grounded Language Learning (Thomason et al., RoboNLP 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-3/W17-2803.pdf
PDF Search