Abstract
Recently, self-learning methods based on user satisfaction metrics and contextual bandits have shown promising results to enable consistent improvements in conversational AI systems. However, directly targeting such metrics by off-policy bandit learning objectives often increases the risk of making abrupt policy changes that break the current user experience. In this study, we introduce a scalable framework for supporting fine-grained exploration targets for individual domains via user-defined constraints. For example, we may want to ensure fewer policy deviations in business-critical domains such as shopping, while allocating more exploration budget to domains such as music. We present a novel meta-gradient learning approach that is scalable and practical to address this problem. The proposed method adjusts constraint violation penalty terms adaptively through a meta objective that encourages balanced constraint satisfaction across domains. We conducted extensive experiments on a real-world conversational AI and using a set of realistic constraint benchmarks. The proposed approach has been deployed in production for a large-scale commercial assistant, enabling the best balance between the policy value and constraint satisfaction rate.- Anthology ID:
- 2023.acl-industry.5
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 43–52
- Language:
- URL:
- https://aclanthology.org/2023.acl-industry.5
- DOI:
- Cite (ACL):
- Mohammad Kachuee and Sungjin Lee. 2023. Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 43–52, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Constrained Policy Optimization for Controlled Self-Learning in Conversational AI Systems (Kachuee & Lee, ACL 2023)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2023.acl-industry.5.pdf