A Simple and Effective Method for Injecting Word-Level Information into Character-Aware Neural Language Models
Yukun Feng, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura
Abstract
We propose a simple and effective method to inject word-level information into character-aware neural language models. Unlike previous approaches which usually inject word-level information at the input of a long short-term memory (LSTM) network, we inject it into the softmax function. The resultant model can be seen as a combination of character-aware language model and simple word-level language model. Our injection method can also be used together with previous methods. Through the experiments on 14 typologically diverse languages, we empirically show that our injection method, when used together with the previous methods, works better than the previous methods, including a gating mechanism, averaging, and concatenation of word vectors. We also provide a comprehensive comparison of these injection methods.- Anthology ID:
- K19-1086
- Volume:
- Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Venue:
- CoNLL
- SIG:
- SIGNLL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 920–928
- Language:
- URL:
- https://aclanthology.org/K19-1086
- DOI:
- 10.18653/v1/K19-1086
- Cite (ACL):
- Yukun Feng, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2019. A Simple and Effective Method for Injecting Word-Level Information into Character-Aware Neural Language Models. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 920–928, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- A Simple and Effective Method for Injecting Word-Level Information into Character-Aware Neural Language Models (Feng et al., CoNLL 2019)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/K19-1086.pdf