Alternative Objective Functions for Training MT Evaluation Metrics

Miloš Stanojević, Khalil Sima’an


Abstract
MT evaluation metrics are tested for correlation with human judgments either at the sentence- or the corpus-level. Trained metrics ignore corpus-level judgments and are trained for high sentence-level correlation only. We show that training only for one objective (sentence or corpus level), can not only harm the performance on the other objective, but it can also be suboptimal for the objective being optimized. To this end we present a metric trained for corpus-level and show empirical comparison against a metric trained for sentence-level exemplifying how their performance may vary per language pair, type and level of judgment. Subsequently we propose a model trained to optimize both objectives simultaneously and show that it is far more stable than–and on average outperforms–both models on both objectives.
Anthology ID:
P17-2004
Volume:
Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2017
Address:
Vancouver, Canada
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
20–25
Language:
URL:
https://aclanthology.org/P17-2004
DOI:
10.18653/v1/P17-2004
Bibkey:
Cite (ACL):
Miloš Stanojević and Khalil Sima’an. 2017. Alternative Objective Functions for Training MT Evaluation Metrics. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20–25, Vancouver, Canada. Association for Computational Linguistics.
Cite (Informal):
Alternative Objective Functions for Training MT Evaluation Metrics (Stanojević & Sima’an, ACL 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/P17-2004.pdf
Video:
 https://vimeo.com/234951559