Just Ask! Evaluating Machine Translation by Asking and Answering Questions
Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, Pavel Pecina
Abstract
In this paper, we show that automatically-generated questions and answers can be used to evaluate the quality of Machine Translation (MT) systems. Building on recent work on the evaluation of abstractive text summarization, we propose a new metric for system-level MT evaluation, compare it with other state-of-the-art solutions, and show its robustness by conducting experiments for various MT directions.- Anthology ID:
- 2021.wmt-1.58
- Volume:
- Proceedings of the Sixth Conference on Machine Translation
- Month:
- November
- Year:
- 2021
- Address:
- Online
- Editors:
- Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussa, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Tom Kocmi, Andre Martins, Makoto Morishita, Christof Monz
- Venue:
- WMT
- SIG:
- SIGMT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 495–506
- Language:
- URL:
- https://aclanthology.org/2021.wmt-1.58
- DOI:
- Cite (ACL):
- Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. 2021. Just Ask! Evaluating Machine Translation by Asking and Answering Questions. In Proceedings of the Sixth Conference on Machine Translation, pages 495–506, Online. Association for Computational Linguistics.
- Cite (Informal):
- Just Ask! Evaluating Machine Translation by Asking and Answering Questions (Krubiński et al., WMT 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2021.wmt-1.58.pdf
- Code
- ufal/mteqa
- Data
- SQuAD, XQuAD