Abstract
End-to-end neural approaches are becoming increasingly common in conversational scenarios due to their promising performances when provided with sufficient amount of data. In this paper, we present a novel methodology to address the interpretability of neural approaches in such scenarios by creating challenge datasets using dialogue self-play over multiple tasks/intents. Dialogue self-play allows generating large amount of synthetic data; by taking advantage of the complete control over the generation process, we show how neural approaches can be evaluated in terms of unseen dialogue patterns. We propose several out-of-pattern test cases each of which introduces a natural and unexpected user utterance phenomenon. As a proof of concept, we built a single and a multiple memory network, and show that these two architectures have diverse performances depending on the peculiar dialogue patterns.- Anthology ID:
- R19-1081
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
- Month:
- September
- Year:
- 2019
- Address:
- Varna, Bulgaria
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 693–702
- Language:
- URL:
- https://aclanthology.org/R19-1081
- DOI:
- 10.26615/978-954-452-056-4_081
- Cite (ACL):
- Sourabh Majumdar, Serra Sinem Tekiroglu, and Marco Guerini. 2019. Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 693–702, Varna, Bulgaria. INCOMA Ltd..
- Cite (Informal):
- Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play (Majumdar et al., RANLP 2019)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/R19-1081.pdf