Are the Tools up to the Task? an Evaluation of Commercial Dialog Tools in Developing Conversational Enterprise-grade Dialog Systems
Marie Meteer, Meghan Hickey, Carmi Rothberg, David Nahamoo, Ellen Eide Kislal
Abstract
There has been a significant investment in dialog systems (tools and runtime) for building conversational systems by major companies including Google, IBM, Microsoft, and Amazon. The question remains whether these tools are up to the task of building conversational, task-oriented dialog applications at the enterprise level. In our company, we are exploring and comparing several toolsets in an effort to determine their strengths and weaknesses in meeting our goals for dialog system development: accuracy, time to market, ease of replicating and extending applications, and efficiency and ease of use by developers. In this paper, we provide both quantitative and qualitative results in three main areas: natural language understanding, dialog, and text generation. While existing toolsets were all incomplete, we hope this paper will provide a roadmap of where they need to go to meet the goal of building effective dialog systems.- Anthology ID:
- N19-2014
- Volume:
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers)
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Anastassia Loukina, Michelle Morales, Rohit Kumar
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 106–113
- Language:
- URL:
- https://aclanthology.org/N19-2014
- DOI:
- 10.18653/v1/N19-2014
- Cite (ACL):
- Marie Meteer, Meghan Hickey, Carmi Rothberg, David Nahamoo, and Ellen Eide Kislal. 2019. Are the Tools up to the Task? an Evaluation of Commercial Dialog Tools in Developing Conversational Enterprise-grade Dialog Systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 106–113, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- Are the Tools up to the Task? an Evaluation of Commercial Dialog Tools in Developing Conversational Enterprise-grade Dialog Systems (Meteer et al., NAACL 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/N19-2014.pdf