A Context-Aware Annotation Framework for Customer Support Live Chat Machine Translation
Miguel Menezes, M. Amin Farajian, Helena Moniz, João Varelas Graça
Abstract
To measure context-aware machine translation (MT) systems quality, existing solutions have recommended human annotators to consider the full context of a document. In our work, we revised a well known Machine Translation quality assessment framework, Multidimensional Quality Metrics (MQM), (Lommel et al., 2014) by introducing a set of nine annotation categories that allows to map MT errors to source document contextual phenomenon, for simplicity sake we named such phenomena as contextual triggers. Our analysis shows that the adapted categories set enhanced MQM’s potential for MT error identification, being able to cover up to 61% more errors, when compared to traditional non-context core MQM’s application. Subsequently, we analyzed the severity of these MT “contextual errors”, showing that the majority fall under the critical and major levels, further indicating the impact of such errors. Finally, we measured the ability of existing evaluation metrics in detecting the proposed MT “contextual errors”. The results have shown that current state-of-the-art metrics fall short in detecting MT errors that are caused by contextual triggers on the source document side. With the work developed, we hope to understand how impactful context is for enhancing quality within a MT workflow and draw attention to future integration of the proposed contextual annotation framework into current MQM’s core typology.- Anthology ID:
- 2023.mtsummit-research.24
- Volume:
- Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track
- Month:
- September
- Year:
- 2023
- Address:
- Macau SAR, China
- Editors:
- Masao Utiyama, Rui Wang
- Venue:
- MTSummit
- SIG:
- Publisher:
- Asia-Pacific Association for Machine Translation
- Note:
- Pages:
- 286–297
- Language:
- URL:
- https://aclanthology.org/2023.mtsummit-research.24
- DOI:
- Cite (ACL):
- Miguel Menezes, M. Amin Farajian, Helena Moniz, and João Varelas Graça. 2023. A Context-Aware Annotation Framework for Customer Support Live Chat Machine Translation. In Proceedings of Machine Translation Summit XIX, Vol. 1: Research Track, pages 286–297, Macau SAR, China. Asia-Pacific Association for Machine Translation.
- Cite (Informal):
- A Context-Aware Annotation Framework for Customer Support Live Chat Machine Translation (Menezes et al., MTSummit 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.mtsummit-research.24.pdf