Abstract
Existing approaches for evaluating word order in machine translation work with metrics computed directly over a permutation of word positions in system output relative to a reference translation. However, every permutation factorizes into a permutation tree (PET) built of primal permutations, i.e., atomic units that do not factorize any further. In this paper we explore the idea that permutations factorizing into (on average) shorter primal permutations should represent simpler ordering as well. Consequently, we contribute Permutation Complexity, a class of metrics over PETs and their extension to forests, and define tight metrics, a sub-class of metrics implementing this idea. Subsequently we define example tight metrics and empirically test them in word order evaluation. Experiments on the WMT13 data sets for ten language pairs show that a tight metric is more often than not better than the baselines.- Anthology ID:
- C16-1204
- Volume:
- Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
- Month:
- December
- Year:
- 2016
- Address:
- Osaka, Japan
- Editors:
- Yuji Matsumoto, Rashmi Prasad
- Venue:
- COLING
- SIG:
- Publisher:
- The COLING 2016 Organizing Committee
- Note:
- Pages:
- 2164–2173
- Language:
- URL:
- https://aclanthology.org/C16-1204
- DOI:
- Cite (ACL):
- Miloš Stanojević and Khalil Sima’an. 2016. Hierarchical Permutation Complexity for Word Order Evaluation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2164–2173, Osaka, Japan. The COLING 2016 Organizing Committee.
- Cite (Informal):
- Hierarchical Permutation Complexity for Word Order Evaluation (Stanojević & Sima’an, COLING 2016)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/C16-1204.pdf