Automating Template Creation for Ranking-Based Dialogue Models
Jingxiang Chen, Heba Elfardy, Simi Wang, Andrea Kahn, Jared Kramer
Abstract
Dialogue response generation models that use template ranking rather than direct sequence generation allow model developers to limit generated responses to pre-approved messages. However, manually creating templates is time-consuming and requires domain expertise. To alleviate this problem, we explore automating the process of creating dialogue templates by using unsupervised methods to cluster historical utterances and selecting representative utterances from each cluster. Specifically, we propose an end-to-end model called Deep Sentence Encoder Clustering (DSEC) that uses an auto-encoder structure to jointly learn the utterance representation and construct template clusters. We compare this method to a random baseline that randomly assigns templates to clusters as well as a strong baseline that performs the sentence encoding and the utterance clustering sequentially. To evaluate the performance of the proposed method, we perform an automatic evaluation with two annotated customer service datasets to assess clustering effectiveness, and a human-in-the-loop experiment using a live customer service application to measure the acceptance rate of the generated templates. DSEC performs best in the automatic evaluation, beats both the sequential and random baselines on most metrics in the human-in-the-loop experiment, and shows promising results when compared to gold/manually created templates.- Anthology ID:
- 2020.nlp4convai-1.9
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Tsung-Hsien Wen, Asli Celikyilmaz, Zhou Yu, Alexandros Papangelis, Mihail Eric, Anuj Kumar, Iñigo Casanueva, Rushin Shah
- Venue:
- NLP4ConvAI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 71–78
- Language:
- URL:
- https://aclanthology.org/2020.nlp4convai-1.9
- DOI:
- 10.18653/v1/2020.nlp4convai-1.9
- Cite (ACL):
- Jingxiang Chen, Heba Elfardy, Simi Wang, Andrea Kahn, and Jared Kramer. 2020. Automating Template Creation for Ranking-Based Dialogue Models. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 71–78, Online. Association for Computational Linguistics.
- Cite (Informal):
- Automating Template Creation for Ranking-Based Dialogue Models (Chen et al., NLP4ConvAI 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.nlp4convai-1.9.pdf