NoDeeLe: A Novel Deep Learning Schema for Evaluating Neural Machine Translation Systems
Despoina Mouratidis, Maria Stasimioti, Vilelmini Sosoni, Katia Lida Kermanidis
Abstract
Due to the wide-spread development of Machine Translation (MT) systems –especially Neural Machine Translation (NMT) systems– MT evaluation, both automatic and human, has become more and more important as it helps us establish how MT systems perform. Yet, automatic evaluation metrics have lagged behind, as the most popular choices (e.g., BLEU, METEOR and ROUGE) may correlate poorly with human judgments. This paper seeks to put to the test an evaluation model based on a novel deep learning schema (NoDeeLe) used to compare two NMT systems on four different text genres, i.e. medical, legal, marketing and literary in the English-Greek language pair. The model utilizes information from the source segments, the MT outputs and the reference translation, as well as the automatic metrics BLEU, METEOR and WER. The proposed schema achieves a strong correlation with human judgment (78% average accuracy for the four texts with the highest accuracy, i.e. 85%, observed in the case of the marketing text), while it outperforms classic machine learning algorithms and automatic metrics.- Anthology ID:
- 2021.triton-1.5
- Volume:
- Proceedings of the Translation and Interpreting Technology Online Conference
- Month:
- July
- Year:
- 2021
- Address:
- Held Online
- Editors:
- Ruslan Mitkov, Vilelmini Sosoni, Julie Christine Giguère, Elena Murgolo, Elizabeth Deysel
- Venue:
- TRITON
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 37–47
- Language:
- URL:
- https://aclanthology.org/2021.triton-1.5
- DOI:
- Cite (ACL):
- Despoina Mouratidis, Maria Stasimioti, Vilelmini Sosoni, and Katia Lida Kermanidis. 2021. NoDeeLe: A Novel Deep Learning Schema for Evaluating Neural Machine Translation Systems. In Proceedings of the Translation and Interpreting Technology Online Conference, pages 37–47, Held Online. INCOMA Ltd..
- Cite (Informal):
- NoDeeLe: A Novel Deep Learning Schema for Evaluating Neural Machine Translation Systems (Mouratidis et al., TRITON 2021)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/2021.triton-1.5.pdf