Towards the evaluation of automatic simultaneous speech translation from a communicative perspective
Abstract
In recent years, automatic speech-to-speech and speech-to-text translation has gained momentum thanks to advances in artificial intelligence, especially in the domains of speech recognition and machine translation. The quality of such applications is commonly tested with automatic metrics, such as BLEU, primarily with the goal of assessing improvements of releases or in the context of evaluation campaigns. However, little is known about how the output of such systems is perceived by end users or how they compare to human performances in similar communicative tasks. In this paper, we present the results of an experiment aimed at evaluating the quality of a real-time speech translation engine by comparing it to the performance of professional simultaneous interpreters. To do so, we adopt a framework developed for the assessment of human interpreters and use it to perform a manual evaluation on both human and machine performances. In our sample, we found better performance for the human interpreters in terms of intelligibility, while the machine performs slightly better in terms of informativeness. The limitations of the study and the possible enhancements of the chosen framework are discussed. Despite its intrinsic limitations, the use of this framework represents a first step towards a user-centric and communication-oriented methodology for evaluating real-time automatic speech translation.- Anthology ID:
- 2021.iwslt-1.29
- Volume:
- Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)
- Month:
- August
- Year:
- 2021
- Address:
- Bangkok, Thailand (online)
- Editors:
- Marcello Federico, Alex Waibel, Marta R. Costa-jussà, Jan Niehues, Sebastian Stuker, Elizabeth Salesky
- Venue:
- IWSLT
- SIG:
- SIGSLT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 245–254
- Language:
- URL:
- https://aclanthology.org/2021.iwslt-1.29
- DOI:
- 10.18653/v1/2021.iwslt-1.29
- Cite (ACL):
- Claudio Fantinuoli and Bianca Prandi. 2021. Towards the evaluation of automatic simultaneous speech translation from a communicative perspective. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 245–254, Bangkok, Thailand (online). Association for Computational Linguistics.
- Cite (Informal):
- Towards the evaluation of automatic simultaneous speech translation from a communicative perspective (Fantinuoli & Prandi, IWSLT 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.iwslt-1.29.pdf