Abstract
Understanding and executing natural language instructions in a grounded domain is one of the hallmarks of artificial intelligence. In this paper, we focus on instruction understanding in the blocks world domain and investigate the language understanding abilities of two top-performing systems for the task. We aim to understand if the test performance of these models indicates an understanding of the spatial domain and of the natural language instructions relative to it, or whether they merely over-fit spurious signals in the dataset. We formulate a set of expectations one might have from an instruction following model and concretely characterize the different dimensions of robustness such a model should possess. Despite decent test performance, we find that state-of-the-art models fall short of these expectations and are extremely brittle. We then propose a learning strategy that involves data augmentation and show through extensive experiments that the proposed learning strategy yields models that are competitive on the original test set while satisfying our expectations much better.- Anthology ID:
- 2021.naacl-main.76
- Volume:
- Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 976–981
- Language:
- URL:
- https://aclanthology.org/2021.naacl-main.76
- DOI:
- 10.18653/v1/2021.naacl-main.76
- Cite (ACL):
- Soham Dan, Michael Zhou, and Dan Roth. 2021. Generalization in Instruction Following Systems. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 976–981, Online. Association for Computational Linguistics.
- Cite (Informal):
- Generalization in Instruction Following Systems (Dan et al., NAACL 2021)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2021.naacl-main.76.pdf