Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs
Maria Stasimioti, Vilelmini Sosoni, Katia Kermanidis, Despoina Mouratidis
Abstract
The present study aims to compare three systems: a generic statistical machine translation (SMT), a generic neural machine translation (NMT) and a tailored-NMT system focusing on the English to Greek language pair. The comparison is carried out following a mixed-methods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post editing (PE) effort and human error analysis performed by 16 postgraduate Translation students. The findings reveal a higher score for both the generic NMT and the tailored-NMT outputs as regards automatic metrics and human evaluation metrics, with the tailored-NMT output faring even better than the generic NMT output.- Anthology ID:
- 2020.eamt-1.47
- Volume:
- Proceedings of the 22nd Annual Conference of the European Association for Machine Translation
- Month:
- November
- Year:
- 2020
- Address:
- Lisboa, Portugal
- Editors:
- André Martins, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, Mikel L. Forcada
- Venue:
- EAMT
- SIG:
- Publisher:
- European Association for Machine Translation
- Note:
- Pages:
- 441–450
- Language:
- URL:
- https://aclanthology.org/2020.eamt-1.47
- DOI:
- Cite (ACL):
- Maria Stasimioti, Vilelmini Sosoni, Katia Kermanidis, and Despoina Mouratidis. 2020. Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 441–450, Lisboa, Portugal. European Association for Machine Translation.
- Cite (Informal):
- Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs (Stasimioti et al., EAMT 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.eamt-1.47.pdf