Abstract
Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable.- Anthology ID:
- 2021.nlp4convai-1.9
- Volume:
- Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI
- Month:
- November
- Year:
- 2021
- Address:
- Online
- Editors:
- Alexandros Papangelis, Paweł Budzianowski, Bing Liu, Elnaz Nouri, Abhinav Rastogi, Yun-Nung Chen
- Venue:
- NLP4ConvAI
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 93–101
- Language:
- URL:
- https://aclanthology.org/2021.nlp4convai-1.9
- DOI:
- 10.18653/v1/2021.nlp4convai-1.9
- Cite (ACL):
- James D. Finch, Sarah E. Finch, and Jinho D. Choi. 2021. What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts. In Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI, pages 93–101, Online. Association for Computational Linguistics.
- Cite (Informal):
- What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts (Finch et al., NLP4ConvAI 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.nlp4convai-1.9.pdf
- Data
- Topical-Chat