AB/BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy
Raphael Petegrosso, VasistaKrishna Baderdinnni, Thibaud Senechal, Benjamin Bullough
Abstract
Evaluation of keyword spotting (KWS) systems that detect keywords in speech is a challenging task under realistic privacy constraints. The KWS is designed to only collect data when the keyword is present, limiting the availability of hard samples that may contain false negatives, and preventing direct estimation of model recall from production data. Alternatively, complementary data collected from other sources may not be fully representative of the real application. In this work, we propose an evaluation technique which we call AB/BA analysis. Our framework evaluates a candidate KWS model B against a baseline model A, using cross-dataset offline decoding for relative recall estimation, without requiring negative examples. Moreover, we propose a formulation with assumptions that allow estimation of relative false positive rate between models with low variance even when the number of false positives is small. Finally, we propose to leverage machine-generated soft labels, in a technique we call Semi-Supervised AB/BA analysis, that improves the analysis time, privacy, and cost. Experiments with both simulation and real data show that AB/BA analysis is successful at measuring recall improvement in conjunction with the trade-off in relative false positive rate.- Anthology ID:
- 2022.naacl-industry.4
- Volume:
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track
- Month:
- July
- Year:
- 2022
- Address:
- Hybrid: Seattle, Washington + Online
- Editors:
- Anastassia Loukina, Rashmi Gangadharaiah, Bonan Min
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 27–36
- Language:
- URL:
- https://aclanthology.org/2022.naacl-industry.4
- DOI:
- 10.18653/v1/2022.naacl-industry.4
- Cite (ACL):
- Raphael Petegrosso, VasistaKrishna Baderdinnni, Thibaud Senechal, and Benjamin Bullough. 2022. AB/BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track, pages 27–36, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
- Cite (Informal):
- AB/BA analysis: A framework for estimating keyword spotting recall improvement while maintaining audio privacy (Petegrosso et al., NAACL 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2022.naacl-industry.4.pdf