Abstract
An important aspect of machine translation is its evaluation, which can be achieved through the use of a variety of metrics. To compare these metrics, the workshop on statistical machine translation annually evaluates metrics based on their correlation with human judgement. Over the years, methods for measuring correlation with humans have changed, but little research has been performed on what the optimal methods for acquiring human scores are and how human correlation can be measured. In this work, the methods for evaluating metrics at both system- and segment-level are analyzed in detail and their shortcomings are pointed out.- Anthology ID:
- 2020.wmt-1.103
- Volume:
- Proceedings of the Fifth Conference on Machine Translation
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- WMT
- SIG:
- SIGMT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 928–933
- Language:
- URL:
- https://aclanthology.org/2020.wmt-1.103
- DOI:
- Cite (ACL):
- Peter Stanchev, Weiyue Wang, and Hermann Ney. 2020. Towards a Better Evaluation of Metrics for Machine Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 928–933, Online. Association for Computational Linguistics.
- Cite (Informal):
- Towards a Better Evaluation of Metrics for Machine Translation (Stanchev et al., WMT 2020)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2020.wmt-1.103.pdf