State Gradients for RNN Memory Analysis
Lyan Verwimp, Hugo Van hamme, Vincent Renkens, Patrick Wambacq
Abstract
We present a framework for analyzing what the state in RNNs remembers from its input embeddings. We compute the gradients of the states with respect to the input embeddings and decompose the gradient matrix with Singular Value Decomposition to analyze which directions in the embedding space are best transferred to the hidden state space, characterized by the largest singular values. We apply our approach to LSTM language models and investigate to what extent and for how long certain classes of words are remembered on average for a certain corpus. Additionally, the extent to which a specific property or relationship is remembered by the RNN can be tracked by comparing a vector characterizing that property with the direction(s) in embedding space that are best preserved in hidden state space.- Anthology ID:
- W18-5443
- Volume:
- Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
- Month:
- November
- Year:
- 2018
- Address:
- Brussels, Belgium
- Editors:
- Tal Linzen, Grzegorz Chrupała, Afra Alishahi
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 344–346
- Language:
- URL:
- https://aclanthology.org/W18-5443
- DOI:
- 10.18653/v1/W18-5443
- Cite (ACL):
- Lyan Verwimp, Hugo Van hamme, Vincent Renkens, and Patrick Wambacq. 2018. State Gradients for RNN Memory Analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 344–346, Brussels, Belgium. Association for Computational Linguistics.
- Cite (Informal):
- State Gradients for RNN Memory Analysis (Verwimp et al., EMNLP 2018)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/W18-5443.pdf
- Data
- Penn Treebank