STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants
Leon Zhang, Jiarui Lu, Joel Ruben Antony Moniz, Aditya Kulkarni, Dhivya Piraviperumal, Tien Dung Tran, Nick Tzou, Hong Yu
Abstract
In the context of a voice assistant system, steering refers to the phenomenon in which a user issues a follow-up command attempting to direct or clarify a previous turn. We propose STEER, a steering detection model that predicts whether a follow-up turn is a user’s attempt to steer the previous command. Constructing a training dataset for steering use cases poses challenges due to the cold-start problem. To overcome this, we developed heuristic rules to sample opt-in usage data, approximating positive and negative samples without any annotation. Our experimental results show promising performance in identifying steering intent, with over 95% accuracy on our sampled data. Moreover, STEER, in conjunction with our sampling strategy, aligns effectively with real-world steering scenarios, as evidenced by its strong zero-shot performance on a human-graded evaluation set. In addition to relying solely on user transcripts as input, we introduce STEER+, an enhanced version of the model. STEER+ utilizes a semantic parse tree to provide more context on out-of-vocabulary words, such as named entities that often occur at the sentence boundary. This further improves model performance, reducing error rate in domains where entities frequently appear, such as messaging. Lastly, we present a data analysis that highlights the improvement in user experience when voice assistants support steering use cases.- Anthology ID:
- 2023.emnlp-industry.61
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Mingxuan Wang, Imed Zitouni
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 640–649
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-industry.61
- DOI:
- 10.18653/v1/2023.emnlp-industry.61
- Cite (ACL):
- Leon Zhang, Jiarui Lu, Joel Ruben Antony Moniz, Aditya Kulkarni, Dhivya Piraviperumal, Tien Dung Tran, Nick Tzou, and Hong Yu. 2023. STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 640–649, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- STEER: Semantic Turn Extension-Expansion Recognition for Voice Assistants (Zhang et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2023.emnlp-industry.61.pdf