Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems
Sarthak Ahuja, Mohammad Kachuee, Fatemeh Sheikholeslami, Weiqing Liu, Jaeyoung Do
Abstract
Off-Policy reinforcement learning has been the driving force for the state-of-the-art conversational AIs leading to more natural human-agent interactions and improving the user satisfaction for goal-oriented agents. However, in large-scale commercial settings, it is often challenging to balance between policy improvements and experience continuity on the broad spectrum of applications handled by such system. In the literature, off-policy evaluation and guard-railing on aggregate statistics has been commonly used to address this problem. In this paper, we propose method for curating and leveraging high-precision samples sourced from historical regression incident reports to validate, safe-guard, and improve policies prior to the online deployment. We conducted extensive experiments using data from a real-world conversational system and actual regression incidents. The proposed method is currently deployed in our production system to protect customers against broken experiences and enable long-term policy improvements.- Anthology ID:
- 2023.acl-industry.35
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Sunayana Sitaram, Beata Beigman Klebanov, Jason D Williams
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 361–367
- Language:
- URL:
- https://aclanthology.org/2023.acl-industry.35
- DOI:
- 10.18653/v1/2023.acl-industry.35
- Cite (ACL):
- Sarthak Ahuja, Mohammad Kachuee, Fatemeh Sheikholeslami, Weiqing Liu, and Jaeyoung Do. 2023. Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 361–367, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Scalable and Safe Remediation of Defective Actions in Self-Learning Conversational Systems (Ahuja et al., ACL 2023)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/2023.acl-industry.35.pdf