Do online Machine Translation Systems Care for Context? What About a GPT Model?

Sheila Castilho, Clodagh Quinn Mallon, Rahel Meister, Shengya Yue

Abstract
This paper addresses the challenges of evaluating document-level machine translation (MT) in the context of recent advances in context-aware neural machine translation (NMT). It investigates how well online MT systems deal with six context-related issues, namely lexical ambiguity, grammatical gender, grammatical number, reference, ellipsis, and terminology, when a larger context span containing the solution for those issues is given as input. Results are compared to the translation outputs from the online ChatGPT. Our results show that, while the change of punctuation in the input yields great variability in the output translations, the context position does not seem to have a great impact. Moreover, the GPT model seems to outperform the NMT systems but performs poorly for Irish. The study aims to provide insights into the effectiveness of online MT systems in handling context and highlight the importance of considering contextual factors in evaluating MT systems.
Anthology ID:
2023.eamt-1.39
Volume:
Proceedings of the 24th Annual Conference of the European Association for Machine Translation
Month:
June
Year:
2023
Address:
Tampere, Finland
Editors:
Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Vanmassenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, Helena Moniz
Venue:
EAMT
SIG:
Publisher:
European Association for Machine Translation
Note:
Pages:
393–417
Language:
URL:
https://aclanthology.org/2023.eamt-1.39
DOI:
Bibkey:
Cite (ACL):
Sheila Castilho, Clodagh Quinn Mallon, Rahel Meister, and Shengya Yue. 2023. Do online Machine Translation Systems Care for Context? What About a GPT Model?. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 393–417, Tampere, Finland. European Association for Machine Translation.
Cite (Informal):
Do online Machine Translation Systems Care for Context? What About a GPT Model? (Castilho et al., EAMT 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/2023.eamt-1.39.pdf
PDF Search