SLTEV: Comprehensive Evaluation of Spoken Language Translation

Ebrahim Ansari, Ondřej Bojar, Barry Haddow, Mohammad Mahmoudi

Abstract
Automatic evaluation of Machine Translation (MT) quality has been investigated over several decades. Spoken Language Translation (SLT), esp. when simultaneous, needs to consider additional criteria and does not have a standard evaluation procedure and a widely used toolkit. To fill the gap, we develop SLTev, an open-source tool for assessing SLT in a comprehensive way. SLTev reports the quality, latency, and stability of an SLT candidate output based on the time-stamped transcript and reference translation into a target language. For quality, we rely on sacreBLEU which provides MT evaluation measures such as chrF or BLEU. For latency, we propose two new scoring techniques. For stability, we extend the previously defined measures with a normalized Flicker in our work. We also propose a new averaging of older measures. A preliminary version of SLTev was used in the IWSLT 2020 shared task. Moreover, a growing collection of test datasets directly accessible by SLTev are provided for system evaluation comparable across papers.
Anthology ID:
2021.eacl-demos.9
Volume:
Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations
Month:
April
Year:
2021
Address:
Online
Editors:
Dimitra Gkatzia, Djamé Seddah
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
71–79
Language:
URL:
https://aclanthology.org/2021.eacl-demos.9
DOI:
10.18653/v1/2021.eacl-demos.9
Bibkey:
Cite (ACL):
Ebrahim Ansari, Ondřej Bojar, Barry Haddow, and Mohammad Mahmoudi. 2021. SLTEV: Comprehensive Evaluation of Spoken Language Translation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 71–79, Online. Association for Computational Linguistics.
Cite (Informal):
SLTEV: Comprehensive Evaluation of Spoken Language Translation (Ansari et al., EACL 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/2021.eacl-demos.9.pdf
Code
 elitr/sltev
PDF Search Code