DIASER: A Unifying View On Task-oriented Dialogue Annotation
Vojtěch Hudeček, Léon-Paul Schaub, Daniel Stancl, Patrick Paroubek, Ondřej Dušek
Abstract
Every model is only as strong as the data that it is trained on. In this paper, we present a new dataset, obtained by merging four publicly available annotated corpora for task-oriented dialogues in several domains (MultiWOZ 2.2, CamRest676, DSTC2 and Schema-Guided Dialogue Dataset). This way, we assess the feasibility of providing a unified ontology and annotation schema covering several domains with a relatively limited effort. We analyze the characteristics of the resulting dataset along three main dimensions: language, information content and performance. We focus on aspects likely to be pertinent for improving dialogue success, e.g. dialogue consistency. Furthermore, to assess the usability of this new corpus, we thoroughly evaluate dialogue generation performance under various conditions with the help of two prominent recent end-to-end dialogue models: MarCo and GPT-2. These models were selected as popular open implementations representative of the two main dimensions of dialogue modelling. While we did not observe a significant gain for dialogue state tracking performance, we show that using more training data from different sources can improve language modelling capabilities and positively impact dialogue flow (consistency). In addition, we provide the community with one of the largest open dataset for machine learning experiments.- Anthology ID:
- 2022.lrec-1.137
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 1286–1296
- Language:
- URL:
- https://aclanthology.org/2022.lrec-1.137
- DOI:
- Cite (ACL):
- Vojtěch Hudeček, Léon-Paul Schaub, Daniel Stancl, Patrick Paroubek, and Ondřej Dušek. 2022. DIASER: A Unifying View On Task-oriented Dialogue Annotation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1286–1296, Marseille, France. European Language Resources Association.
- Cite (Informal):
- DIASER: A Unifying View On Task-oriented Dialogue Annotation (Hudeček et al., LREC 2022)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2022.lrec-1.137.pdf
- Code
- ufal/diaser
- Data
- SGD