Evaluation of Machine Translation with Predictive Metrics beyond BLEU/NIST: CESTA Evaluation Campaign # 1
Sylvain Surcin, Olivier Hamon, Antony Hartley, Martin Rajman, Andrei Popescu-Belis, Widad Mustafa El Hadi, Ismaïl Timimi, Marianne Dabbadie, Khalid Choukri
Abstract
In this paper, we report on the results of a full-size evaluation campaign of various MT systems. This campaign is novel compared to the classical DARPA/NIST MT evaluation campaigns in the sense that French is the target language, and that it includes an experiment of meta-evaluation of various metrics claiming to better predict different attributes of translation quality. We first describe the campaign, its context, its protocol and the data we used. Then we summarise the results obtained by the participating systems and discuss the meta-evaluation of the metrics used.- Anthology ID:
- 2005.mtsummit-papers.16
- Volume:
- Proceedings of Machine Translation Summit X: Papers
- Month:
- September 13-15
- Year:
- 2005
- Address:
- Phuket, Thailand
- Venue:
- MTSummit
- SIG:
- Publisher:
- Note:
- Pages:
- 117–124
- Language:
- URL:
- https://aclanthology.org/2005.mtsummit-papers.16
- DOI:
- Cite (ACL):
- Sylvain Surcin, Olivier Hamon, Antony Hartley, Martin Rajman, Andrei Popescu-Belis, Widad Mustafa El Hadi, Ismaïl Timimi, Marianne Dabbadie, and Khalid Choukri. 2005. Evaluation of Machine Translation with Predictive Metrics beyond BLEU/NIST: CESTA Evaluation Campaign # 1. In Proceedings of Machine Translation Summit X: Papers, pages 117–124, Phuket, Thailand.
- Cite (Informal):
- Evaluation of Machine Translation with Predictive Metrics beyond BLEU/NIST: CESTA Evaluation Campaign # 1 (Surcin et al., MTSummit 2005)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2005.mtsummit-papers.16.pdf