Abstract
This work describes analysis of nature and causes of MT errors observed by different evaluators under guidance of different quality criteria: adequacy and comprehension and and a not specified generic mixture of adequacy and fluency. We report results for three language pairs and two domains and eleven MT systems. Our findings indicate that and despite the fact that some of the identified phenomena depend on domain and/or language and the following set of phenomena can be considered as generally challenging for modern MT systems: rephrasing groups of words and translation of ambiguous source words and translating noun phrases and and mistranslations. Furthermore and we show that the quality criterion also has impact on error perception. Our findings indicate that comprehension and adequacy can be assessed simultaneously by different evaluators and so that comprehension and as an important quality criterion and can be included more often in human evaluations.- Anthology ID:
- 2021.mtsummit-research.14
- Volume:
- Proceedings of Machine Translation Summit XVIII: Research Track
- Month:
- August
- Year:
- 2021
- Address:
- Virtual
- Editors:
- Kevin Duh, Francisco Guzmán
- Venue:
- MTSummit
- SIG:
- Publisher:
- Association for Machine Translation in the Americas
- Note:
- Pages:
- 163–175
- Language:
- URL:
- https://aclanthology.org/2021.mtsummit-research.14
- DOI:
- Cite (ACL):
- Maja Popovic. 2021. On nature and causes of observed MT errors. In Proceedings of Machine Translation Summit XVIII: Research Track, pages 163–175, Virtual. Association for Machine Translation in the Americas.
- Cite (Informal):
- On nature and causes of observed MT errors (Popovic, MTSummit 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2021.mtsummit-research.14.pdf
- Code
- awslabs/sockeye