Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency

Shigeki Karita, Richard Sproat, Haruko Ishikawa

Abstract
Word error rate (WER) and character error rate (CER) are standard metrics in Speech Recognition (ASR), but one problem has always been alternative spellings: If one’s system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word. Japanese is notorious for “lacking orthography”: most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation. In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR. We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana. In a manual evaluation, raters rated 95.4% of the proposed spelling variants as plausible. ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%–3.1% absolute reduction in CER depending on the task.
Anthology ID:
2023.cawl-1.8
Volume:
Proceedings of the Workshop on Computation and Written Language (CAWL 2023)
Month:
July
Year:
2023
Address:
Toronto, Canada
Editors:
Kyle Gorman, Richard Sproat, Brian Roark
Venue:
CAWL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
61–70
Language:
URL:
https://aclanthology.org/2023.cawl-1.8
DOI:
10.18653/v1/2023.cawl-1.8
Bibkey:
Cite (ACL):
Shigeki Karita, Richard Sproat, and Haruko Ishikawa. 2023. Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 61–70, Toronto, Canada. Association for Computational Linguistics.
Cite (Informal):
Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency (Karita et al., CAWL 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/dois-2013-emnlp/2023.cawl-1.8.pdf
PDF Search