Input-to-Output Gate to Improve RNN Language Models

Sho Takase, Jun Suzuki, Masaaki Nagata


Abstract
This paper proposes a reinforcing method that refines the output layers of existing Recurrent Neural Network (RNN) language models. We refer to our proposed method as Input-to-Output Gate (IOG). IOG has an extremely simple structure, and thus, can be easily combined with any RNN language models. Our experiments on the Penn Treebank and WikiText-2 datasets demonstrate that IOG consistently boosts the performance of several different types of current topline RNN language models.
Anthology ID:
I17-2008
Volume:
Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers)
Month:
November
Year:
2017
Address:
Taipei, Taiwan
Venue:
IJCNLP
SIG:
Publisher:
Asian Federation of Natural Language Processing
Note:
Pages:
43–48
Language:
URL:
https://aclanthology.org/I17-2008
DOI:
Bibkey:
Cite (ACL):
Sho Takase, Jun Suzuki, and Masaaki Nagata. 2017. Input-to-Output Gate to Improve RNN Language Models. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 43–48, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Cite (Informal):
Input-to-Output Gate to Improve RNN Language Models (Takase et al., IJCNLP 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/I17-2008.pdf
Code
 nttcslab-nlp/iog
Data
Penn TreebankWikiText-2