Using Morphological Knowledge in Open-Vocabulary Neural Language Models

Austin Matthews, Graham Neubig, Chris Dyer

Abstract
Languages with productive morphology pose problems for language models that generate words from a fixed vocabulary. Although character-based models allow any possible word type to be generated, they are linguistically naïve: they must discover that words exist and are delimited by spaces—basic linguistic facts that are built in to the structure of word-based models. We introduce an open-vocabulary language model that incorporates more sophisticated linguistic knowledge by predicting words using a mixture of three generative processes: (1) by generating words as a sequence of characters, (2) by directly generating full word forms, and (3) by generating words as a sequence of morphemes that are combined using a hand-written morphological analyzer. Experiments on Finnish, Turkish, and Russian show that our model outperforms character sequence models and other strong baselines on intrinsic and extrinsic measures. Furthermore, we show that our model learns to exploit morphological knowledge encoded in the analyzer, and, as a byproduct, it can perform effective unsupervised morphological disambiguation.
Anthology ID:
N18-1130
Volume:
Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
Month:
June
Year:
2018
Address:
New Orleans, Louisiana
Editors:
Marilyn Walker, Heng Ji, Amanda Stent
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1435–1445
Language:
URL:
https://aclanthology.org/N18-1130
DOI:
10.18653/v1/N18-1130
Bibkey:
Cite (ACL):
Austin Matthews, Graham Neubig, and Chris Dyer. 2018. Using Morphological Knowledge in Open-Vocabulary Neural Language Models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1435–1445, New Orleans, Louisiana. Association for Computational Linguistics.
Cite (Informal):
Using Morphological Knowledge in Open-Vocabulary Neural Language Models (Matthews et al., NAACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-2/N18-1130.pdf
PDF Search