Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs

Maria Stasimioti, Vilelmini Sosoni, Katia Kermanidis, Despoina Mouratidis


Abstract
The present study aims to compare three systems: a generic statistical machine translation (SMT), a generic neural machine translation (NMT) and a tailored-NMT system focusing on the English to Greek language pair. The comparison is carried out following a mixed-methods approach, i.e. automatic metrics, as well as side-by-side ranking, adequacy and fluency rating, measurement of actual post editing (PE) effort and human error analysis performed by 16 postgraduate Translation students. The findings reveal a higher score for both the generic NMT and the tailored-NMT outputs as regards automatic metrics and human evaluation metrics, with the tailored-NMT output faring even better than the generic NMT output.
Anthology ID:
2020.eamt-1.47
Volume:
Proceedings of the 22nd Annual Conference of the European Association for Machine Translation
Month:
November
Year:
2020
Address:
Lisboa, Portugal
Venue:
EAMT
SIG:
Publisher:
European Association for Machine Translation
Note:
Pages:
441–450
Language:
URL:
https://aclanthology.org/2020.eamt-1.47
DOI:
Bibkey:
Cite (ACL):
Maria Stasimioti, Vilelmini Sosoni, Katia Kermanidis, and Despoina Mouratidis. 2020. Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 441–450, Lisboa, Portugal. European Association for Machine Translation.
Cite (Informal):
Machine Translation Quality: A comparative evaluation of SMT, NMT and tailored-NMT outputs (Stasimioti et al., EAMT 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/2020.eamt-1.47.pdf