Abstract
Model churn occurs when re-training a model yields different predictions despite using the same data and hyper-parameters. Churn reduction is crucial for industry conversational systems where users expect consistent results for the same queries. In this setting, compute resources are often limited due to latency requirements during serving and overall time constraints during re-training. To address this issue, we propose a compute-efficient method that mitigates churn without requiring extra resources for training or inference. Our approach involves a lightweight data pre-processing step that pairs semantic parses based on their “function call signature” and encourages similarity through an additional loss based on Jensen-Shannon Divergence. We validate the effectiveness of our method in three scenarios: academic (+3.93 percent improvement on average in a churn reduction metric), simulated noisy data (+8.09), and industry (+5.28) settings.- Anthology ID:
- 2023.emnlp-industry.28
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Mingxuan Wang, Imed Zitouni
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 284–293
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-industry.28
- DOI:
- 10.18653/v1/2023.emnlp-industry.28
- Cite (ACL):
- Christopher Hidey and Sarthak Sarthak. 2023. Compute-Efficient Churn Reduction for Conversational Agents. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 284–293, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Compute-Efficient Churn Reduction for Conversational Agents (Hidey & Sarthak, EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2023.emnlp-industry.28.pdf