Improving BERT Pretraining with Syntactic Supervision
Georgios Tziafas, Konstantinos Kogkalidis, Gijs Wijnholds, Michael Moortgat
Abstract
Bidirectional masked Transformers have become the core theme in the current NLP landscape. Despite their impressive benchmarks, a recurring theme in recent research has been to question such models’ capacity for syntactic generalization. In this work, we seek to address this question by adding a supervised, token-level supertagging objective to standard unsupervised pretraining, enabling the explicit incorporation of syntactic biases into the network’s training dynamics. Our approach is straightforward to implement, induces a marginal computational overhead and is general enough to adapt to a variety of settings. We apply our methodology on Lassy Large, an automatically annotated corpus of written Dutch. Our experiments suggest that our syntax-aware model performs on par with established baselines, despite Lassy Large being one order of magnitude smaller than commonly used corpora.- Anthology ID:
- 2023.clasp-1.18
- Volume:
- Proceedings of the 2023 CLASP Conference on Learning with Small Data (LSD)
- Month:
- September
- Year:
- 2023
- Address:
- Gothenburg, Sweden
- Editors:
- Ellen Breitholtz, Shalom Lappin, Sharid Loaiciga, Nikolai Ilinykh, Simon Dobnik
- Venue:
- CLASP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 176–184
- Language:
- URL:
- https://aclanthology.org/2023.clasp-1.18
- DOI:
- Cite (ACL):
- Georgios Tziafas, Konstantinos Kogkalidis, Gijs Wijnholds, and Michael Moortgat. 2023. Improving BERT Pretraining with Syntactic Supervision. In Proceedings of the 2023 CLASP Conference on Learning with Small Data (LSD), pages 176–184, Gothenburg, Sweden. Association for Computational Linguistics.
- Cite (Informal):
- Improving BERT Pretraining with Syntactic Supervision (Tziafas et al., CLASP 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-07/2023.clasp-1.18.pdf