Abstract
MT evaluation metrics are tested for correlation with human judgments either at the sentence- or the corpus-level. Trained metrics ignore corpus-level judgments and are trained for high sentence-level correlation only. We show that training only for one objective (sentence or corpus level), can not only harm the performance on the other objective, but it can also be suboptimal for the objective being optimized. To this end we present a metric trained for corpus-level and show empirical comparison against a metric trained for sentence-level exemplifying how their performance may vary per language pair, type and level of judgment. Subsequently we propose a model trained to optimize both objectives simultaneously and show that it is far more stable than–and on average outperforms–both models on both objectives.- Anthology ID:
- P17-2004
- Volume:
- Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2017
- Address:
- Vancouver, Canada
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 20–25
- Language:
- URL:
- https://aclanthology.org/P17-2004
- DOI:
- 10.18653/v1/P17-2004
- Cite (ACL):
- Miloš Stanojević and Khalil Sima’an. 2017. Alternative Objective Functions for Training MT Evaluation Metrics. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20–25, Vancouver, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Alternative Objective Functions for Training MT Evaluation Metrics (Stanojević & Sima’an, ACL 2017)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/P17-2004.pdf