Revisiting the Hierarchical Multiscale LSTM
Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, Afra Alishahi
Abstract
Hierarchical Multiscale LSTM (Chung et. al., 2016) is a state-of-the-art language model that learns interpretable structure from character-level input. Such models can provide fertile ground for (cognitive) computational linguistics studies. However, the high complexity of the architecture, training and implementations might hinder its applicability. We provide a detailed reproduction and ablation study of the architecture, shedding light on some of the potential caveats of re-purposing complex deep-learning architectures. We further show that simplifying certain aspects of the architecture can in fact improve its performance. We also investigate the linguistic units (segments) learned by various levels of the model, and argue that their quality does not correlate with the overall performance of the model on language modeling.- Anthology ID:
- C18-1272
- Volume:
- Proceedings of the 27th International Conference on Computational Linguistics
- Month:
- August
- Year:
- 2018
- Address:
- Santa Fe, New Mexico, USA
- Editors:
- Emily M. Bender, Leon Derczynski, Pierre Isabelle
- Venue:
- COLING
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3215–3227
- Language:
- URL:
- https://aclanthology.org/C18-1272
- DOI:
- Cite (ACL):
- Ákos Kádár, Marc-Alexandre Côté, Grzegorz Chrupała, and Afra Alishahi. 2018. Revisiting the Hierarchical Multiscale LSTM. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3215–3227, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
- Cite (Informal):
- Revisiting the Hierarchical Multiscale LSTM (Kádár et al., COLING 2018)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/C18-1272.pdf
- Data
- Penn Treebank