Abstract
Human language is known to exhibit a nested, hierarchical structure, allowing us to form complex sentences out of smaller pieces. However, many state-of-the-art neural networks models such as Transformers have no explicit hierarchical structure in their architecture—that is, they don’t have an inductive bias toward hierarchical structure. Additionally, Transformers are known to perform poorly on compositional generalization tasks which require such structures. In this paper, we introduce Treeformer, a general-purpose encoder module inspired by the CKY algorithm which learns a composition operator and pooling function to construct hierarchical encodings for phrases and sentences. Our extensive experiments demonstrate the benefits of incorporating hierarchical structure into the Transformer and show significant improvements in compositional generalization as well as in downstream tasks such as machine translation, abstractive summarization, and various natural language understanding tasks.- Anthology ID:
- 2023.ranlp-1.90
- Volume:
- Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing
- Month:
- September
- Year:
- 2023
- Address:
- Varna, Bulgaria
- Editors:
- Ruslan Mitkov, Galia Angelova
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd., Shoumen, Bulgaria
- Note:
- Pages:
- 836–845
- Language:
- URL:
- https://aclanthology.org/2023.ranlp-1.90
- DOI:
- Cite (ACL):
- Nilay Patel and Jeffrey Flanigan. 2023. Forming Trees with Treeformers. In Proceedings of the 14th International Conference on Recent Advances in Natural Language Processing, pages 836–845, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
- Cite (Informal):
- Forming Trees with Treeformers (Patel & Flanigan, RANLP 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/2023.ranlp-1.90.pdf