Abstract
In this paper, we propose a new approach to employ the fixed-size ordinally-forgetting encoding (FOFE) (Zhang et al., 2015b) in neural languages modelling, called dual-FOFE. The main idea of dual-FOFE is that it allows to use two different forgetting factors so that it can avoid the trade-off in choosing either a small or large values for the single forgetting factor. In our experiments, we have compared the dual-FOFE based neural network language models (NNLM) against the original FOFE counterparts and various traditional NNLMs. Our results on the challenging Google Billion word corpus show that both FOFE and dual FOFE yield very strong performance while significantly reducing the computational complexity over other NNLMs. Furthermore, the proposed dual-FOFE method further gives over 10% improvement in perplexity over the original FOFE model.- Anthology ID:
- D18-1502
- Volume:
- Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing
- Month:
- October-November
- Year:
- 2018
- Address:
- Brussels, Belgium
- Editors:
- Ellen Riloff, David Chiang, Julia Hockenmaier, Jun’ichi Tsujii
- Venue:
- EMNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4725–4730
- Language:
- URL:
- https://aclanthology.org/D18-1502
- DOI:
- 10.18653/v1/D18-1502
- Cite (ACL):
- Sedtawut Watcharawittayakul, Mingbin Xu, and Hui Jiang. 2018. Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4725–4730, Brussels, Belgium. Association for Computational Linguistics.
- Cite (Informal):
- Dual Fixed-Size Ordinally Forgetting Encoding (FOFE) for Competitive Neural Language Models (Watcharawittayakul et al., EMNLP 2018)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/D18-1502.pdf