CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues

Francisco Javier Chiyah Garcia, José Lopes, Xingkun Liu, Helen Hastie

Abstract
Large corpora of task-based and open-domain conversational dialogues are hugely valuable in the field of data-driven dialogue systems. Crowdsourcing platforms, such as Amazon Mechanical Turk, have been an effective method for collecting such large amounts of data. However, difficulties arise when task-based dialogues require expert domain knowledge or rapid access to domain-relevant information, such as databases for tourism. This will become even more prevalent as dialogue systems become increasingly ambitious, expanding into tasks with high levels of complexity that require collaboration and forward planning, such as in our domain of emergency response. In this paper, we propose CRWIZ: a framework for collecting real-time Wizard of Oz dialogues through crowdsourcing for collaborative, complex tasks. This framework uses semi-guided dialogue to avoid interactions that breach procedures and processes only known to experts, while enabling the capture of a wide variety of interactions.
Anthology ID:
2020.lrec-1.36
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
288–297
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.36
DOI:
Bibkey:
Cite (ACL):
Francisco Javier Chiyah Garcia, José Lopes, Xingkun Liu, and Helen Hastie. 2020. CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 288–297, Marseille, France. European Language Resources Association.
Cite (Informal):
CRWIZ: A Framework for Crowdsourcing Real-Time Wizard-of-Oz Dialogues (Chiyah Garcia et al., LREC 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/2020.lrec-1.36.pdf
Code
 JChiyah/crwiz
Data
MultiWOZ
PDF Search Code