Probing for targeted syntactic knowledge through grammatical error detection
Christopher Davis, Christopher Bryant, Andrew Caines, Marek Rei, Paula Buttery
Abstract
Targeted studies testing knowledge of subject-verb agreement (SVA) indicate that pre-trained language models encode syntactic information. We assert that if models robustly encode subject-verb agreement, they should be able to identify when agreement is correct and when it is incorrect. To that end, we propose grammatical error detection as a diagnostic probe to evaluate token-level contextual representations for their knowledge of SVA. We evaluate contextual representations at each layer from five pre-trained English language models: BERT, XLNet, GPT-2, RoBERTa and ELECTRA. We leverage public annotated training data from both English second language learners and Wikipedia edits, and report results on manually crafted stimuli for subject-verb agreement. We find that masked language models linearly encode information relevant to the detection of SVA errors, while the autoregressive models perform on par with our baseline. However, we also observe a divergence in performance when probes are trained on different training sets, and when they are evaluated on different syntactic constructions, suggesting the information pertaining to SVA error detection is not robustly encoded.- Anthology ID:
- 2022.conll-1.25
- Volume:
- Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Antske Fokkens, Vivek Srikumar
- Venue:
- CoNLL
- SIG:
- SIGNLL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 360–373
- Language:
- URL:
- https://aclanthology.org/2022.conll-1.25
- DOI:
- 10.18653/v1/2022.conll-1.25
- Cite (ACL):
- Christopher Davis, Christopher Bryant, Andrew Caines, Marek Rei, and Paula Buttery. 2022. Probing for targeted syntactic knowledge through grammatical error detection. In Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 360–373, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Probing for targeted syntactic knowledge through grammatical error detection (Davis et al., CoNLL 2022)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2022.conll-1.25.pdf