Goal-Oriented End-to-End Conversational Models with Profile Features in a Real-World Setting
Yichao Lu, Manisha Srivastava, Jared Kramer, Heba Elfardy, Andrea Kahn, Song Wang, Vikas Bhardwaj
Abstract
End-to-end neural models for goal-oriented conversational systems have become an increasingly active area of research, though results in real-world settings are few. We present real-world results for two issue types in the customer service domain. We train models on historical chat transcripts and test on live contacts using a human-in-the-loop research platform. Additionally, we incorporate customer profile features to assess their impact on model performance. We experiment with two approaches for response generation: (1) sequence-to-sequence generation and (2) template ranking. To test our models, a customer service agent handles live contacts and at each turn we present the top four model responses and allow the agent to select (and optionally edit) one of the suggestions or to type their own. We present results for turn acceptance rate, response coverage, and edit rate based on approximately 600 contacts, as well as qualitative analysis on patterns of turn rejection and edit behavior. Top-4 turn acceptance rate across all models ranges from 63%-80%. Our results suggest that these models are promising for an agent-support application.- Anthology ID:
- N19-2007
- Volume:
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers)
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Anastassia Loukina, Michelle Morales, Rohit Kumar
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 48–55
- Language:
- URL:
- https://aclanthology.org/N19-2007
- DOI:
- 10.18653/v1/N19-2007
- Cite (ACL):
- Yichao Lu, Manisha Srivastava, Jared Kramer, Heba Elfardy, Andrea Kahn, Song Wang, and Vikas Bhardwaj. 2019. Goal-Oriented End-to-End Conversational Models with Profile Features in a Real-World Setting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 48–55, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- Goal-Oriented End-to-End Conversational Models with Profile Features in a Real-World Setting (Lu et al., NAACL 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/N19-2007.pdf