BETOLD: A Task-Oriented Dialog Dataset for Breakdown Detection

Silvia Terragni, Bruna Guedes, Andre Manso, Modestas Filipavicius, Nghia Khau, Roland Mathis


Abstract
Task-Oriented Dialog (TOD) systems often suffer from dialog breakdowns - situations in which users cannot or do not want to proceed with the conversation. Ideally TOD systems should be able to detect dialog breakdowns to prevent users from quitting a conversation and to encourage them to interact with the system again. In this paper, we present BETOLD, a privacy-preserving dataset for breakdown detection. The dataset consists of user and system turns represented by intents and entity annotations, derived from NLU and NLG dialog manager components. We also propose an attention-based model that detects potential breakdowns using these annotations, instead of the utterances’ text. This approach achieves a comparable performance to the corresponding utterance-only model, while ensuring data privacy.
Anthology ID:
2022.cai-1.4
Volume:
Proceedings of the Second Workshop on When Creative AI Meets Conversational AI
Month:
October
Year:
2022
Address:
Gyeongju, Republic of Korea
Venue:
CAI
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
23–34
Language:
URL:
https://aclanthology.org/2022.cai-1.4
DOI:
Bibkey:
Cite (ACL):
Silvia Terragni, Bruna Guedes, Andre Manso, Modestas Filipavicius, Nghia Khau, and Roland Mathis. 2022. BETOLD: A Task-Oriented Dialog Dataset for Breakdown Detection. In Proceedings of the Second Workshop on When Creative AI Meets Conversational AI, pages 23–34, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Cite (Informal):
BETOLD: A Task-Oriented Dialog Dataset for Breakdown Detection (Terragni et al., CAI 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.cai-1.4.pdf