Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences
Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, Yoshua Bengio
Abstract
We propose a hierarchical model for sequential data that learns a tree on-the-fly, i.e. while reading the sequence. In the model, a recurrent network adapts its structure and reuses recurrent weights in a recursive manner. This creates adaptive skip-connections that ease the learning of long-term dependencies. The tree structure can either be inferred without supervision through reinforcement learning, or learned in a supervised manner. We provide preliminary experiments in a novel Math Expression Evaluation (MEE) task, which is created to have a hierarchical tree structure that can be used to study the effectiveness of our model. Additionally, we test our model in a well-known propositional logic and language modelling tasks. Experimental results have shown the potential of our approach.- Anthology ID:
- W18-3020
- Volume:
- Proceedings of the Third Workshop on Representation Learning for NLP
- Month:
- July
- Year:
- 2018
- Address:
- Melbourne, Australia
- Venue:
- RepL4NLP
- SIG:
- SIGREP
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 154–158
- Language:
- URL:
- https://aclanthology.org/W18-3020
- DOI:
- 10.18653/v1/W18-3020
- Cite (ACL):
- Athul Paul Jacob, Zhouhan Lin, Alessandro Sordoni, and Yoshua Bengio. 2018. Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences. In Proceedings of the Third Workshop on Representation Learning for NLP, pages 154–158, Melbourne, Australia. Association for Computational Linguistics.
- Cite (Informal):
- Learning Hierarchical Structures On-The-Fly with a Recurrent-Recursive Model for Sequences (Jacob et al., RepL4NLP 2018)
- PDF:
- https://preview.aclanthology.org/starsem-semeval-split/W18-3020.pdf