Abstract
A line of work on Transformer-based language models such as BERT has attempted to use syntactic inductive bias to enhance the pretraining process, on the theory that building syntactic structure into the training process should reduce the amount of data needed for training. But such methods are often tested for high-resource languages such as English. In this work, we investigate whether these methods can compensate for data sparseness in low-resource languages, hypothesizing that they ought to be more effective for low-resource languages. We experiment with five low-resource languages: Uyghur, Wolof, Maltese, Coptic, and Ancient Greek. We find that these syntactic inductive bias methods produce uneven results in low-resource settings, and provide surprisingly little benefit in most cases.- Anthology ID:
- 2023.conll-1.17
- Volume:
- Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL)
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Jing Jiang, David Reitter, Shumin Deng
- Venue:
- CoNLL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 238–253
- Language:
- URL:
- https://aclanthology.org/2023.conll-1.17
- DOI:
- 10.18653/v1/2023.conll-1.17
- Cite (ACL):
- Luke Gessler and Nathan Schneider. 2023. Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages?. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 238–253, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Syntactic Inductive Bias in Transformer Language Models: Especially Helpful for Low-Resource Languages? (Gessler & Schneider, CoNLL 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.conll-1.17.pdf