Train, Sort, Explain: Learning to Diagnose Translation Models
Robert Schwarzenberg, David Harbecke, Vivien Macketanz, Eleftherios Avramidis, Sebastian Möller
Abstract
Evaluating translation models is a trade-off between effort and detail. On the one end of the spectrum there are automatic count-based methods such as BLEU, on the other end linguistic evaluations by humans, which arguably are more informative but also require a disproportionately high effort. To narrow the spectrum, we propose a general approach on how to automatically expose systematic differences between human and machine translations to human experts. Inspired by adversarial settings, we train a neural text classifier to distinguish human from machine translations. A classifier that performs and generalizes well after training should recognize systematic differences between the two classes, which we uncover with neural explainability methods. Our proof-of-concept implementation, DiaMaT, is open source. Applied to a dataset translated by a state-of-the-art neural Transformer model, DiaMaT achieves a classification accuracy of 75% and exposes meaningful differences between humans and the Transformer, amidst the current discussion about human parity.- Anthology ID:
- N19-4006
- Volume:
- Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, Minnesota
- Editors:
- Waleed Ammar, Annie Louis, Nasrin Mostafazadeh
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 29–34
- Language:
- URL:
- https://aclanthology.org/N19-4006
- DOI:
- 10.18653/v1/N19-4006
- Cite (ACL):
- Robert Schwarzenberg, David Harbecke, Vivien Macketanz, Eleftherios Avramidis, and Sebastian Möller. 2019. Train, Sort, Explain: Learning to Diagnose Translation Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pages 29–34, Minneapolis, Minnesota. Association for Computational Linguistics.
- Cite (Informal):
- Train, Sort, Explain: Learning to Diagnose Translation Models (Schwarzenberg et al., NAACL 2019)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/N19-4006.pdf
- Code
- dfki-nlp/diamat
- Data
- WMT 2014