Linguistically Motivated Evaluation of Machine Translation Metrics Based on a Challenge Set

Eleftherios Avramidis, Vivien Macketanz


Abstract
We employ a linguistically motivated challenge set in order to evaluate the state-of-the-art machine translation metrics submitted to the Metrics Shared Task of the 7th Conference for Machine Translation. The challenge set includes about 20,000 items extracted from 145 MT systems for two language directions (German-English, English-German), covering more than 100 linguistically-motivated phenomena organized in 14 categories. The best performing metrics are YiSi-1, BERTScore and COMET-22 for German-English, and UniTE, UniTE-ref, XL-DA and xxl-DA19 for English-German.Metrics in both directions are performing worst when it comes to named-entities & terminology and particularly measuring units. Particularly in German-English they are weak at detecting issues at punctuation, polar questions, relative clauses, dates and idioms. In English-German, they perform worst at present progressive of transitive verbs, future II progressive of intransitive verbs, simple present perfect of ditransitive verbs and focus particles.
Anthology ID:
2022.wmt-1.45
Volume:
Proceedings of the Seventh Conference on Machine Translation (WMT)
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates (Hybrid)
Editors:
Philipp Koehn, Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Kocmi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, Marcos Zampieri
Venue:
WMT
SIG:
SIGMT
Publisher:
Association for Computational Linguistics
Note:
Pages:
514–529
Language:
URL:
https://aclanthology.org/2022.wmt-1.45
DOI:
Bibkey:
Cite (ACL):
Eleftherios Avramidis and Vivien Macketanz. 2022. Linguistically Motivated Evaluation of Machine Translation Metrics Based on a Challenge Set. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 514–529, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Cite (Informal):
Linguistically Motivated Evaluation of Machine Translation Metrics Based on a Challenge Set (Avramidis & Macketanz, WMT 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2022.wmt-1.45.pdf