Abstract
As natural language technology becomes ever-present in everyday life, people will expect artificial agents to understand language use as humans do. Nevertheless, most advanced neural AI systems fail at some types of interactions that are trivial for humans (e.g., ask a smart system “What am I pointing at?”). One critical aspect of human language understanding is situated reasoning, where inferences make reference to the local context, perceptual surroundings, and contextual groundings from the interaction. In this cutting-edge tutorial, we bring to the NLP/CL community a synthesis of multimodal grounding and meaning representation techniques with formal and computational models of embodied reasoning. We will discuss existing approaches to multimodal language grounding and meaning representations, discuss the kind of information each method captures and their relative suitability to situated reasoning tasks, and demon- strate how to construct agents that conduct situated reasoning by embodying a simulated environment. In doing so, these agents also represent their human interlocutor(s) within the simulation, and are represented through their virtual embodiment in the real world, enabling true bidirectional communication with a computer using multiple modalities.- Anthology ID:
- 2022.aacl-tutorials.4
- Volume:
- Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts
- Month:
- November
- Year:
- 2022
- Address:
- Taipei
- Venues:
- AACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 22–27
- Language:
- URL:
- https://aclanthology.org/2022.aacl-tutorials.4
- DOI:
- Cite (ACL):
- Nikhil Krishnaswamy and James Pustejovsky. 2022. Grounding Meaning Representation for Situated Reasoning. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts, pages 22–27, Taipei. Association for Computational Linguistics.
- Cite (Informal):
- Grounding Meaning Representation for Situated Reasoning (Krishnaswamy & Pustejovsky, AACL-IJCNLP 2022)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2022.aacl-tutorials.4.pdf