Abstract
Long short-term memory (LSTM) language model (LM) has been widely investigated for automatic speech recognition (ASR) and natural language processing (NLP). Although excellent performance is obtained for large vocabulary tasks, tremendous memory consumption prohibits the use of LSTM LM in low-resource devices. The memory consumption mainly comes from the word embedding layer. In this paper, a novel binarized LSTM LM is proposed to address the problem. Words are encoded into binary vectors and other LSTM parameters are further binarized to achieve high memory compression. This is the first effort to investigate binary LSTM for large vocabulary LM. Experiments on both English and Chinese LM and ASR tasks showed that can achieve a compression ratio of 11.3 without any loss of LM and ASR performances and a compression ratio of 31.6 with acceptable minor performance degradation.- Anthology ID:
- N18-1192
- Volume:
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Marilyn Walker, Heng Ji, Amanda Stent
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2113–2121
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/N18-1192/
- DOI:
- 10.18653/v1/N18-1192
- Cite (ACL):
- Xuan Liu, Di Cao, and Kai Yu. 2018. Binarized LSTM Language Model. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2113–2121, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- Binarized LSTM Language Model (Liu et al., NAACL 2018)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/N18-1192.pdf
- Data
- Penn Treebank