Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations
Meiguo Wang, Benjamin Yao, Bin Guo, Xiaohu Liu, Yu Zhang, Tuan-Hung Pham, Chenlei Guo
Abstract
To evaluate the performance of a multi-domain goal-oriented Dialogue System (DS), it is important to understand what the users’ goals are for the conversations and whether those goals are successfully achieved. The success rate of goals directly correlates with user satisfaction and perceived usefulness of the DS. In this paper, we propose a novel automatic dialogue evaluation framework that jointly performs two tasks: goal segmentation and goal success prediction. We extend the RoBERTa-IQ model (Gupta et al., 2021) by adding multi-task learning heads for goal segmentation and success prediction. Using an annotated dataset from a commercial DS, we demonstrate that our proposed model reaches an accuracy that is on-par with single-pass human annotation comparing to a three-pass gold annotation benchmark.- Anthology ID:
- 2022.coling-1.41
- Volume:
- Proceedings of the 29th International Conference on Computational Linguistics
- Month:
- October
- Year:
- 2022
- Address:
- Gyeongju, Republic of Korea
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 505–509
- Language:
- URL:
- https://aclanthology.org/2022.coling-1.41
- DOI:
- Cite (ACL):
- Meiguo Wang, Benjamin Yao, Bin Guo, Xiaohu Liu, Yu Zhang, Tuan-Hung Pham, and Chenlei Guo. 2022. Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 505–509, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
- Cite (Informal):
- Joint Goal Segmentation and Goal Success Prediction on Multi-Domain Conversations (Wang et al., COLING 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.coling-1.41.pdf