Abstract
Multi-modal grounded language learning connects language predicates to physical properties of objects in the world. Sensing with multiple modalities, such as audio, haptics, and visual colors and shapes while performing interaction behaviors like lifting, dropping, and looking on objects enables a robot to ground non-visual predicates like “empty” as well as visual predicates like “red”. Previous work has established that grounding in multi-modal space improves performance on object retrieval from human descriptions. In this work, we gather behavior annotations from humans and demonstrate that these improve language grounding performance by allowing a system to focus on relevant behaviors for words like “white” or “half-full” that can be understood by looking or lifting, respectively. We also explore adding modality annotations (whether to focus on audio or haptics when performing a behavior), which improves performance, and sharing information between linguistically related predicates (if “green” is a color, “white” is a color), which improves grounding recall but at the cost of precision.- Anthology ID:
- W17-2803
- Volume:
- Proceedings of the First Workshop on Language Grounding for Robotics
- Month:
- August
- Year:
- 2017
- Address:
- Vancouver, Canada
- Editors:
- Mohit Bansal, Cynthia Matuszek, Jacob Andreas, Yoav Artzi, Yonatan Bisk
- Venue:
- RoboNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 20–24
- Language:
- URL:
- https://aclanthology.org/W17-2803
- DOI:
- 10.18653/v1/W17-2803
- Cite (ACL):
- Jesse Thomason, Jivko Sinapov, and Raymond Mooney. 2017. Guiding Interaction Behaviors for Multi-modal Grounded Language Learning. In Proceedings of the First Workshop on Language Grounding for Robotics, pages 20–24, Vancouver, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Guiding Interaction Behaviors for Multi-modal Grounded Language Learning (Thomason et al., RoboNLP 2017)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/W17-2803.pdf