Abstract
Previous studies on machine translation evaluation mostly focused on the quality of individual sentences, while overlooking the important role of contextual information. Although WMT Metrics Shared Tasks have introduced context content into the human annotations of translation evaluation since 2019, the relevant metrics and methods still did not take advantage of the corresponding context. In this paper, we propose a context-aware machine translation evaluation metric called Cont-COMET, built upon the effective COMET framework. Our approach simultaneously considers the preceding and subsequent contexts of the sentence to be evaluated and trains our metric to be aligned with the setting during human annotation. We also introduce a content selection method to extract and utilize the most relevant information. The experiments and evaluation of Cont-COMET on the official test framework from WMT show improvements in both system-level and segment-level assessments.- Anthology ID:
- 2023.findings-emnlp.1021
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15291–15298
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.1021
- DOI:
- 10.18653/v1/2023.findings-emnlp.1021
- Cite (ACL):
- Xinyu Hu, Xunjian Yin, and Xiaojun Wan. 2023. Exploring Context-Aware Evaluation Metrics for Machine Translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15291–15298, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Exploring Context-Aware Evaluation Metrics for Machine Translation (Hu et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2023.findings-emnlp.1021.pdf