Abstract
Automatic evaluation comparing candidate translations to human-generated paraphrases of reference translations has recently been proposed by freitag2020bleu. When used in place of original references, the paraphrased versions produce metric scores that correlate better with human judgment. This effect holds for a variety of different automatic metrics, and tends to favor natural formulations over more literal (translationese) ones. In this paper we compare the results of performing end-to-end system development using standard and paraphrased references. With state-of-the-art English-German NMT components, we show that tuning to paraphrased references produces a system that is ignificantly better according to human judgment, but 5 BLEU points worse when tested on standard references. Our work confirms the finding that paraphrased references yield metric scores that correlate better with human judgment, and demonstrates for the first time that using these scores for system development can lead to significant improvements.- Anthology ID:
- 2020.wmt-1.140
- Volume:
- Proceedings of the Fifth Conference on Machine Translation
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri
- Venue:
- WMT
- SIG:
- SIGMT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1183–1192
- Language:
- URL:
- https://aclanthology.org/2020.wmt-1.140
- DOI:
- Cite (ACL):
- Markus Freitag, George Foster, David Grangier, and Colin Cherry. 2020. Human-Paraphrased References Improve Neural Machine Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 1183–1192, Online. Association for Computational Linguistics.
- Cite (Informal):
- Human-Paraphrased References Improve Neural Machine Translation (Freitag et al., WMT 2020)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2020.wmt-1.140.pdf