Abstract
Word error rate (WER) and character error rate (CER) are standard metrics inSpeech Recognition (ASR), but one problem has always been alternative spellings: If one’s system transcribes adviser whereas the ground truth has advisor, this will count as an error even though the two spellings really represent the same word.Japanese is notorious for “lacking orthography”: most words can be spelled in multiple ways, presenting a problem for accurate ASR evaluation. In this paper we propose a new lenient evaluation metric as a more defensible CER measure for Japanese ASR. We create a lattice of plausible respellings of the reference transcription, using a combination of lexical resources, a Japanese text-processing system, and a neural machine translation model for reconstructing kanji from hiragana or katakana. In amanual evaluation, raters rated 95.4% of the proposed spelling variants as plausible. ASR results show that our method, which does not penalize the system for choosing a valid alternate spelling of a word, affords a 2.4%–3.1% absolute reduction in CER depending on the task.- Anthology ID:
- 2023.cawl-1.8
- Volume:
- Proceedings of the Workshop on Computation and Written Language (CAWL 2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Venue:
- CAWL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 61–70
- Language:
- URL:
- https://aclanthology.org/2023.cawl-1.8
- DOI:
- Cite (ACL):
- Shigeki Karita, Richard Sproat, and Haruko Ishikawa. 2023. Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency. In Proceedings of the Workshop on Computation and Written Language (CAWL 2023), pages 61–70, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Lenient Evaluation of Japanese Speech Recognition: Modeling Naturally Occurring Spelling Inconsistency (Karita et al., CAWL 2023)
- PDF:
- https://preview.aclanthology.org/starsem-semeval-split/2023.cawl-1.8.pdf