Abstract
Previous work has established that RNNs with an unbounded activation function have the capacity to count exactly. However, it has also been shown that RNNs are challenging to train effectively and generally do not learn exact counting behaviour. In this paper, we focus on this problem by studying the simplest possible RNN, a linear single-cell network. We conduct a theoretical analysis of linear RNNs and identify conditions for the models to exhibit exact counting behaviour. We provide a formal proof that these conditions are necessary and sufficient. We also conduct an empirical analysis using tasks involving a Dyck-1-like Balanced Bracket language under two different settings. We observe that linear RNNs generally do not meet the necessary and sufficient conditions for counting behaviour when trained with the standard approach. We investigate how varying the length of training sequences and utilising different target classes impacts model behaviour during training and the ability of linear RNN models to effectively approximate the indicator conditions.- Anthology ID:
- 2023.eacl-srw.15
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Elisa Bassignana, Matthias Lindemann, Alban Petit
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 143–148
- Language:
- URL:
- https://aclanthology.org/2023.eacl-srw.15
- DOI:
- 10.18653/v1/2023.eacl-srw.15
- Cite (ACL):
- Nadine El-Naggar, Pranava Madhyastha, and Tillman Weyde. 2023. Theoretical Conditions and Empirical Failure of Bracket Counting on Long Sequences with Linear Recurrent Networks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 143–148, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Theoretical Conditions and Empirical Failure of Bracket Counting on Long Sequences with Linear Recurrent Networks (El-Naggar et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.eacl-srw.15.pdf