Revisiting Round-trip Translation for Quality Estimation

Jihyung Moon, Hyunchang Cho, Eunjeong L. Park


Abstract
Quality estimation (QE), a task of evaluating the quality of translation automatically without human-translated reference, is one of the important challenges for machine translation (MT). As the QE methods, BLEU score for round-trip translation (RTT) had been considered. However, it was found to be a poor predictor of translation quality since BLEU was not an adequate metric to detect semantic similarity between input and RTT. Recently, the pre-trained language models have made breakthroughs in many NLP tasks by providing semantically meaningful word and sentence embeddings. In this paper, we employ the semantic embeddings to RTT-based QE metric. Our method achieves the highest correlations with human judgments compared to WMT 2019 quality estimation metric task submissions. Additionally, we observe that with semantic-level metrics, RTT-based QE is robust to the choice of a backward translation system and shows consistent performance on both SMT and NMT forward translation systems.
Anthology ID:
2020.eamt-1.11
Volume:
Proceedings of the 22nd Annual Conference of the European Association for Machine Translation
Month:
November
Year:
2020
Address:
Lisboa, Portugal
Venue:
EAMT
SIG:
Publisher:
European Association for Machine Translation
Note:
Pages:
91–104
Language:
URL:
https://aclanthology.org/2020.eamt-1.11
DOI:
Bibkey:
Cite (ACL):
Jihyung Moon, Hyunchang Cho, and Eunjeong L. Park. 2020. Revisiting Round-trip Translation for Quality Estimation. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 91–104, Lisboa, Portugal. European Association for Machine Translation.
Cite (Informal):
Revisiting Round-trip Translation for Quality Estimation (Moon et al., EAMT 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.eamt-1.11.pdf
Data
PAWSWMT19 Metrics Task