Challenging the State-of-the-art Machine Translation Metrics from a Linguistic Perspective
Eleftherios Avramidis, Shushen Manakhimova, Vivien Macketanz, Sebastian Möller
Abstract
We employ a linguistically motivated challenge set in order to evaluate the state-of-the-art machine translation metrics submitted to the Metrics Shared Task of the 8th Conference for Machine Translation. The challenge set includes about 21,000 items extracted from 155 machine translation systems for three language directions, covering more than 100 linguistically-motivated phenomena organized in 14 categories. The metrics that have the best performance with regard to our linguistically motivated analysis are the Cometoid22-wmt23 (a trained metric based on distillation) for German-English and MetricX-23-c (based on a fine-tuned mT5 encoder-decoder language model) for English-German and English-Russian. Some of the most difficult phenomena are passive voice for German-English, named entities, terminology and measurement units for English-German, and focus particles, adverbial clause and stripping for English-Russian.- Anthology ID:
- 2023.wmt-1.58
- Volume:
- Proceedings of the Eighth Conference on Machine Translation
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Philipp Koehn, Barry Haddow, Tom Kocmi, Christof Monz
- Venue:
- WMT
- SIG:
- SIGMT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 713–729
- Language:
- URL:
- https://aclanthology.org/2023.wmt-1.58
- DOI:
- 10.18653/v1/2023.wmt-1.58
- Cite (ACL):
- Eleftherios Avramidis, Shushen Manakhimova, Vivien Macketanz, and Sebastian Möller. 2023. Challenging the State-of-the-art Machine Translation Metrics from a Linguistic Perspective. In Proceedings of the Eighth Conference on Machine Translation, pages 713–729, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Challenging the State-of-the-art Machine Translation Metrics from a Linguistic Perspective (Avramidis et al., WMT 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.wmt-1.58.pdf