Perturbation Based Learning for Structured NLP Tasks with Application to Dependency Parsing

Amichay Doitch, Ram Yazdi, Tamir Hazan, Roi Reichart

Abstract
The best solution of structured prediction models in NLP is often inaccurate because of limited expressive power of the model or to non-exact parameter estimation. One way to mitigate this problem is sampling candidate solutions from the model’s solution space, reasoning that effective exploration of this space should yield high-quality solutions. Unfortunately, sampling is often computationally hard and many works hence back-off to sub-optimal strategies, such as extraction of the best scoring solutions of the model, which are not as diverse as sampled solutions. In this paper we propose a perturbation-based approach where sampling from a probabilistic model is computationally efficient. We present a learning algorithm for the variance of the perturbations, and empirically demonstrate its importance. Moreover, while finding the argmax in our model is intractable, we propose an efficient and effective approximation. We apply our framework to cross-lingual dependency parsing across 72 corpora from 42 languages and to lightly supervised dependency parsing across 13 corpora from 12 languages, and demonstrate strong results in terms of both the quality of the entire solution list and of the final solution.1
Anthology ID:
Q19-1041
Volume:
Transactions of the Association for Computational Linguistics, Volume 7
Month:
Year:
2019
Address:
Cambridge, MA
Editors:
Lillian Lee, Mark Johnson, Brian Roark, Ani Nenkova
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
643–659
Language:
URL:
https://aclanthology.org/Q19-1041
DOI:
10.1162/tacl_a_00291
Bibkey:
Cite (ACL):
Amichay Doitch, Ram Yazdi, Tamir Hazan, and Roi Reichart. 2019. Perturbation Based Learning for Structured NLP Tasks with Application to Dependency Parsing. Transactions of the Association for Computational Linguistics, 7:643–659.
Cite (Informal):
Perturbation Based Learning for Structured NLP Tasks with Application to Dependency Parsing (Doitch et al., TACL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ml4al-ingestion/Q19-1041.pdf
PDF Search