Hierarchical Permutation Complexity for Word Order Evaluation

Miloš Stanojević, Khalil Sima’an


Abstract
Existing approaches for evaluating word order in machine translation work with metrics computed directly over a permutation of word positions in system output relative to a reference translation. However, every permutation factorizes into a permutation tree (PET) built of primal permutations, i.e., atomic units that do not factorize any further. In this paper we explore the idea that permutations factorizing into (on average) shorter primal permutations should represent simpler ordering as well. Consequently, we contribute Permutation Complexity, a class of metrics over PETs and their extension to forests, and define tight metrics, a sub-class of metrics implementing this idea. Subsequently we define example tight metrics and empirically test them in word order evaluation. Experiments on the WMT13 data sets for ten language pairs show that a tight metric is more often than not better than the baselines.
Anthology ID:
C16-1204
Volume:
Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers
Month:
December
Year:
2016
Address:
Osaka, Japan
Venue:
COLING
SIG:
Publisher:
The COLING 2016 Organizing Committee
Note:
Pages:
2164–2173
Language:
URL:
https://aclanthology.org/C16-1204
DOI:
Bibkey:
Cite (ACL):
Miloš Stanojević and Khalil Sima’an. 2016. Hierarchical Permutation Complexity for Word Order Evaluation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2164–2173, Osaka, Japan. The COLING 2016 Organizing Committee.
Cite (Informal):
Hierarchical Permutation Complexity for Word Order Evaluation (Stanojević & Sima’an, COLING 2016)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/C16-1204.pdf