Abstract
This paper offers a comparative evaluation of four commercial ASR systems which are evaluated according to the post-editing effort required to reach “publishable” quality and according to the number of errors they produce. For the error annotation task, an original error typology for transcription errors is proposed. This study also seeks to examine whether there is a difference in the performance of these systems between native and non-native English speakers. The experimental results suggest that among the four systems, Trint obtains the best scores. It is also observed that most systems perform noticeably better with native speakers and that all systems are most prone to fluency errors.- Anthology ID:
- 2021.triton-1.23
- Volume:
- Proceedings of the Translation and Interpreting Technology Online Conference
- Month:
- July
- Year:
- 2021
- Address:
- Held Online
- Editors:
- Ruslan Mitkov, Vilelmini Sosoni, Julie Christine Giguère, Elena Murgolo, Elizabeth Deysel
- Venue:
- TRITON
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 199–207
- Language:
- URL:
- https://aclanthology.org/2021.triton-1.23
- DOI:
- Cite (ACL):
- Martha Maria Papadopoulou, Anna Zaretskaya, and Ruslan Mitkov. 2021. Benchmarking ASR Systems Based on Post-Editing Effort and Error Analysis. In Proceedings of the Translation and Interpreting Technology Online Conference, pages 199–207, Held Online. INCOMA Ltd..
- Cite (Informal):
- Benchmarking ASR Systems Based on Post-Editing Effort and Error Analysis (Papadopoulou et al., TRITON 2021)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2021.triton-1.23.pdf