Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture
Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, Manabu Okumura
Abstract
Character-aware neural language models can capture the relationship between words by exploiting character-level information and are particularly effective for languages with rich morphology. However, these models are usually biased towards information from surface forms. To alleviate this problem, we propose a simple and effective method to improve a character-aware neural language model by forcing a character encoder to produce word-based embeddings under Skip-gram architecture in a warm-up step without extra training data. We empirically show that the resulting character-aware neural language model achieves obvious improvements of perplexity scores on typologically diverse languages, that contain many low-frequency or unseen words.- Anthology ID:
- 2021.ranlp-1.48
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
- Month:
- September
- Year:
- 2021
- Address:
- Held Online
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 421–427
- Language:
- URL:
- https://aclanthology.org/2021.ranlp-1.48
- DOI:
- Cite (ACL):
- Yukun Feng, Chenlong Hu, Hidetaka Kamigaito, Hiroya Takamura, and Manabu Okumura. 2021. Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 421–427, Held Online. INCOMA Ltd..
- Cite (Informal):
- Improving Character-Aware Neural Language Model by Warming up Character Encoder under Skip-gram Architecture (Feng et al., RANLP 2021)
- PDF:
- https://preview.aclanthology.org/paclic-22-ingestion/2021.ranlp-1.48.pdf