Abstract
We analyse Recurrent Neural Networks (RNNs) to understand the significance of multiple LSTM layers. We argue that the Weighted Finite-state Automata (WFA) trained using a spectral learning algorithm are helpful to analyse RNNs. Our results suggest that multiple LSTM layers in RNNs help learning distributed hidden states, but have a smaller impact on the ability to learn long-term dependencies. The analysis is based on the empirical results, however relevant theory (whenever possible) was discussed to justify and support our conclusions.- Anthology ID:
- W19-3903
- Volume:
- Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges
- Month:
- August
- Year:
- 2019
- Address:
- Florence
- Editors:
- Jason Eisner, Matthias Gallé, Jeffrey Heinz, Ariadna Quattoni, Guillaume Rabusseau
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 24–33
- Language:
- URL:
- https://aclanthology.org/W19-3903
- DOI:
- 10.18653/v1/W19-3903
- Cite (ACL):
- Farhana Ferdousi Liza and Marek Grzes. 2019. Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 24–33, Florence. Association for Computational Linguistics.
- Cite (Informal):
- Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling (Liza & Grzes, ACL 2019)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/W19-3903.pdf
- Data
- Penn Treebank