Abstract
This paper discusses a classification-based approach to machine translation evaluation, as opposed to a common regression-based approach in the WMT Metrics task. Recent machine translation usually works well but sometimes makes critical errors due to just a few wrong word choices. Our classification-based approach focuses on such errors using several error type labels, for practical machine translation evaluation in an age of neural machine translation. We made additional annotations on the WMT 2015-2017 Metrics datasets with fluency and adequacy labels to distinguish different types of translation errors from syntactic and semantic viewpoints. We present our human evaluation criteria for the corpus development and automatic evaluation experiments using the corpus. The human evaluation corpus will be publicly available upon publication.- Anthology ID:
- 2021.humeval-1.5
- Volume:
- Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Editors:
- Anya Belz, Shubham Agarwal, Yvette Graham, Ehud Reiter, Anastasia Shimorina
- Venue:
- HumEval
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 46–55
- Language:
- URL:
- https://aclanthology.org/2021.humeval-1.5
- DOI:
- Cite (ACL):
- Katsuhito Sudoh, Kosuke Takahashi, and Satoshi Nakamura. 2021. Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors. In Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), pages 46–55, Online. Association for Computational Linguistics.
- Cite (Informal):
- Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors (Sudoh et al., HumEval 2021)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/2021.humeval-1.5.pdf