Abstract
Syntactic information has been proved to be useful for transformer-based pre-trained language models. Previous studies often rely on additional syntax-guided attention components to enhance the transformer, which require more parameters and additional syntactic parsing in downstream tasks. This increase in complexity severely limits the application of syntax-enhanced language model in a wide range of scenarios. In order to inject syntactic knowledge effectively and efficiently into pre-trained language models, we propose a novel syntax-guided contrastive learning method which does not change the transformer architecture. Based on constituency and dependency structures of syntax trees, we design phrase-guided and tree-guided contrastive objectives, and optimize them in the pre-training stage, so as to help the pre-trained language model to capture rich syntactic knowledge in its representations. Experimental results show that our contrastive method achieves consistent improvements in a variety of tasks, including grammatical error detection, entity tasks, structural probing and GLUE. Detailed analysis further verifies that the improvements come from the utilization of syntactic information, and the learned attention weights are more explainable in terms of linguistics.- Anthology ID:
- 2022.findings-acl.191
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2022
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Smaranda Muresan, Preslav Nakov, Aline Villavicencio
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2430–2440
- Language:
- URL:
- https://aclanthology.org/2022.findings-acl.191
- DOI:
- 10.18653/v1/2022.findings-acl.191
- Cite (ACL):
- Shuai Zhang, Wang Lijie, Xinyan Xiao, and Hua Wu. 2022. Syntax-guided Contrastive Learning for Pre-trained Language Model. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2430–2440, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Syntax-guided Contrastive Learning for Pre-trained Language Model (Zhang et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2022.findings-acl.191.pdf
- Data
- BLiMP, CoLA, CoNLL 2003, FCE, GLUE, Open Entity, QNLI