Data Collection for Dialogue System: A Startup Perspective

Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Lingjia Tang, Jason Mars


Abstract
Industrial dialogue systems such as Apple Siri and Google Now rely on large scale diverse and robust training data to enable their sophisticated conversation capability. Crowdsourcing provides a scalable and inexpensive way of data collection but collecting high quality data efficiently requires thoughtful orchestration of the crowdsourcing jobs. Prior study of this topic have focused on tasks only in the academia settings with limited scope or only provide intrinsic dataset analysis, lacking indication on how it affects the trained model performance. In this paper, we present a study of crowdsourcing methods for a user intent classification task in our deployed dialogue system. Our task requires classification of 47 possible user intents and contains many intent pairs with subtle differences. We consider different crowdsourcing job types and job prompts and analyze quantitatively the quality of the collected data and the downstream model performance on a test set of real user queries from production logs. Our observation provides insights into designing efficient crowdsourcing jobs and provide recommendations for future dialogue system data collection process.
Anthology ID:
N18-3005
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)
Month:
June
Year:
2018
Address:
New Orleans - Louisiana
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
33–40
Language:
URL:
https://aclanthology.org/N18-3005
DOI:
10.18653/v1/N18-3005
Bibkey:
Cite (ACL):
Yiping Kang, Yunqi Zhang, Jonathan K. Kummerfeld, Lingjia Tang, and Jason Mars. 2018. Data Collection for Dialogue System: A Startup Perspective. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pages 33–40, New Orleans - Louisiana. Association for Computational Linguistics.
Cite (Informal):
Data Collection for Dialogue System: A Startup Perspective (Kang et al., NAACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/N18-3005.pdf
Video:
 http://vimeo.com/277631102