A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography
Noëmi Aepli, Chantal Amrhein, Florian Schottmann, Rico Sennrich
Abstract
For sensible progress in natural language processing, it is important that we are aware of the limitations of the evaluation metrics we use. In this work, we evaluate how robust metrics are to non-standardized dialects, i.e. spelling differences in language varieties that do not have a standard orthography. To investigate this, we collect a dataset of human translations and human judgments for automatic machine translations from English to two Swiss German dialects. We further create a challenge set for dialect variation and benchmark existing metrics’ performances. Our results show that existing metrics cannot reliably evaluate Swiss German text generation outputs, especially on segment level. We propose initial design adaptations that increase robustness in the face of non-standardized dialects, although there remains much room for further improvement. The dataset, code, and models are available here: https://github.com/textshuttle/dialect_eval- Anthology ID:
- 2023.wmt-1.99
- Volume:
- Proceedings of the Eighth Conference on Machine Translation
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Philipp Koehn, Barry Haddow, Tom Kocmi, Christof Monz
- Venue:
- WMT
- SIG:
- SIGMT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1045–1065
- Language:
- URL:
- https://aclanthology.org/2023.wmt-1.99
- DOI:
- 10.18653/v1/2023.wmt-1.99
- Cite (ACL):
- Noëmi Aepli, Chantal Amrhein, Florian Schottmann, and Rico Sennrich. 2023. A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography. In Proceedings of the Eighth Conference on Machine Translation, pages 1045–1065, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- A Benchmark for Evaluating Machine Translation Metrics on Dialects without Standard Orthography (Aepli et al., WMT 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.wmt-1.99.pdf