Integrated taxonomy of errors in chat-oriented dialogue systems
Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, Masahiro Mizukami
Abstract
This paper proposes a taxonomy of errors in chat-oriented dialogue systems. Previously, two taxonomies were proposed; one is theory-driven and the other data-driven. The former suffers from the fact that dialogue theories for human conversation are often not appropriate for categorizing errors made by chat-oriented dialogue systems. The latter has limitations in that it can only cope with errors of systems for which we have data. This paper integrates these two taxonomies to create a comprehensive taxonomy of errors in chat-oriented dialogue systems. We found that, with our integrated taxonomy, errors can be reliably annotated with a higher Fleiss’ kappa compared with the previously proposed taxonomies.- Anthology ID:
- 2021.sigdial-1.10
- Volume:
- Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue
- Month:
- July
- Year:
- 2021
- Address:
- Singapore and Online
- Editors:
- Haizhou Li, Gina-Anne Levow, Zhou Yu, Chitralekha Gupta, Berrak Sisman, Siqi Cai, David Vandyke, Nina Dethlefs, Yan Wu, Junyi Jessy Li
- Venue:
- SIGDIAL
- SIG:
- SIGDIAL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 89–98
- Language:
- URL:
- https://aclanthology.org/2021.sigdial-1.10
- DOI:
- 10.18653/v1/2021.sigdial-1.10
- Cite (ACL):
- Ryuichiro Higashinaka, Masahiro Araki, Hiroshi Tsukahara, and Masahiro Mizukami. 2021. Integrated taxonomy of errors in chat-oriented dialogue systems. In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 89–98, Singapore and Online. Association for Computational Linguistics.
- Cite (Informal):
- Integrated taxonomy of errors in chat-oriented dialogue systems (Higashinaka et al., SIGDIAL 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2021.sigdial-1.10.pdf
- Code
- ryuichiro-higashinaka/taxonomy-of-errors