Validating Quality Estimation in a Computer-Aided Translation Workflow: Speed, Cost and Quality Trade-off

Fernando Alva-Manchego, Lucia Specia, Sara Szoc, Tom Vanallemeersch, Heidi Depraetere

Abstract
In modern computer-aided translation workflows, Machine Translation (MT) systems are used to produce a draft that is then checked and edited where needed by human translators. In this scenario, a Quality Estimation (QE) tool can be used to score MT outputs, and a threshold on the QE scores can be applied to decide whether an MT output can be used as-is or requires human post-edition. While this could reduce cost and turnaround times, it could harm translation quality, as QE models are not 100% accurate. In the framework of the APE-QUEST project (Automated Post-Editing and Quality Estimation), we set up a case-study on the trade-off between speed, cost and quality, investigating the benefits of QE models in a real-world scenario, where we rely on end-user acceptability as quality metric. Using data in the public administration domain for English-Dutch and English-French, we experimented with two use cases: assimilation and dissemination. Results shed some light on how QE scores can be explored to establish thresholds that suit each use case and target language, and demonstrate the potential benefits of adding QE to a translation workflow.
Anthology ID:
2021.mtsummit-up.22
Volume:
Proceedings of Machine Translation Summit XVIII: Users and Providers Track
Month:
August
Year:
2021
Address:
Virtual
Venue:
MTSummit
SIG:
Publisher:
Association for Machine Translation in the Americas
Note:
Pages:
306–315
Language:
URL:
https://aclanthology.org/2021.mtsummit-up.22
DOI:
Bibkey:
Cite (ACL):
Fernando Alva-Manchego, Lucia Specia, Sara Szoc, Tom Vanallemeersch, and Heidi Depraetere. 2021. Validating Quality Estimation in a Computer-Aided Translation Workflow: Speed, Cost and Quality Trade-off. In Proceedings of Machine Translation Summit XVIII: Users and Providers Track, pages 306–315, Virtual. Association for Machine Translation in the Americas.
Cite (Informal):
Validating Quality Estimation in a Computer-Aided Translation Workflow: Speed, Cost and Quality Trade-off (Alva-Manchego et al., MTSummit 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2021.mtsummit-up.22.pdf
PDF Search