Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling

Farhana Ferdousi Liza, Marek Grzes

[How to correct problems with metadata yourself]


Abstract
We analyse Recurrent Neural Networks (RNNs) to understand the significance of multiple LSTM layers. We argue that the Weighted Finite-state Automata (WFA) trained using a spectral learning algorithm are helpful to analyse RNNs. Our results suggest that multiple LSTM layers in RNNs help learning distributed hidden states, but have a smaller impact on the ability to learn long-term dependencies. The analysis is based on the empirical results, however relevant theory (whenever possible) was discussed to justify and support our conclusions.
Anthology ID:
W19-3903
Volume:
Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges
Month:
August
Year:
2019
Address:
Florence
Editors:
Jason Eisner, Matthias Gallé, Jeffrey Heinz, Ariadna Quattoni, Guillaume Rabusseau
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
24–33
Language:
URL:
https://aclanthology.org/W19-3903
DOI:
10.18653/v1/W19-3903
Bibkey:
Cite (ACL):
Farhana Ferdousi Liza and Marek Grzes. 2019. Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 24–33, Florence. Association for Computational Linguistics.
Cite (Informal):
Relating RNN Layers with the Spectral WFA Ranks in Sequence Modelling (Liza & Grzes, ACL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/teach-a-man-to-fish/W19-3903.pdf
Data
Penn Treebank