It Is Not As Good As You Think! Evaluating Simultaneous Machine Translation on Interpretation Data

Jinming Zhao, Philip Arthur, Gholamreza Haffari, Trevor Cohn, Ehsan Shareghi

Abstract
Most existing simultaneous machine translation (SiMT) systems are trained and evaluated on offline translation corpora. We argue that SiMT systems should be trained and tested on real interpretation data. To illustrate this argument, we propose an interpretation test set and conduct a realistic evaluation of SiMT trained on offline translations. Our results, on our test set along with 3 existing smaller scale language pairs, highlight the difference of up-to 13.83 BLEU score when SiMT models are evaluated on translation vs interpretation data. In the absence of interpretation training data, we propose a translation-to-interpretation (T2I) style transfer method which allows converting existing offline translations into interpretation-style data, leading to up-to 2.8 BLEU improvement. However, the evaluation gap remains notable, calling for constructing large-scale interpretation corpora better suited for evaluating and developing SiMT systems.
Anthology ID:
2021.emnlp-main.537
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6707–6715
Language:
URL:
https://aclanthology.org/2021.emnlp-main.537
DOI:
10.18653/v1/2021.emnlp-main.537
Bibkey:
Cite (ACL):
Jinming Zhao, Philip Arthur, Gholamreza Haffari, Trevor Cohn, and Ehsan Shareghi. 2021. It Is Not As Good As You Think! Evaluating Simultaneous Machine Translation on Interpretation Data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6707–6715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
It Is Not As Good As You Think! Evaluating Simultaneous Machine Translation on Interpretation Data (Zhao et al., EMNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.emnlp-main.537.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2021.emnlp-main.537.mp4
Code
 mingzi151/interpretationdata
Data
Europarl
PDF Search Code Video