Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains

Kallirroi Georgila, Anton Leuski, Volodymyr Yanov, David Traum

Abstract
We evaluate several publicly available off-the-shelf (commercial and research) automatic speech recognition (ASR) systems across diverse dialogue domains (in US-English). Our evaluation is aimed at non-experts with limited experience in speech recognition. Our goal is not only to compare a variety of ASR systems on several diverse data sets but also to measure how much ASR technology has advanced since our previous large-scale evaluations on the same data sets. Our results show that the performance of each speech recognizer can vary significantly depending on the domain. Furthermore, despite major recent progress in ASR technology, current state-of-the-art speech recognizers perform poorly in domains that require special vocabulary and language models, and under noisy conditions. We expect that our evaluation will prove useful to ASR consumers and dialogue system designers.
Anthology ID:
2020.lrec-1.797
Volume:
Proceedings of the Twelfth Language Resources and Evaluation Conference
Month:
May
Year:
2020
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
6469–6476
Language:
English
URL:
https://aclanthology.org/2020.lrec-1.797
DOI:
Bibkey:
Cite (ACL):
Kallirroi Georgila, Anton Leuski, Volodymyr Yanov, and David Traum. 2020. Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6469–6476, Marseille, France. European Language Resources Association.
Cite (Informal):
Evaluation of Off-the-shelf Speech Recognizers Across Diverse Dialogue Domains (Georgila et al., LREC 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_acl24_videos/2020.lrec-1.797.pdf
PDF Search