Abstract
In this paper, we explore how much syntactic supervision is “good enough” to make language models (LMs) more human-like. Specifically, we propose the new method called syntactic ablation, where syntactic LMs, namely Recurrent Neural Network Grammars (RNNGs), are gradually ablated from full syntactic supervision to zero syntactic supervision (≈ unidirectional LSTM) by preserving NP, VP, PP, SBAR nonterminal symbols and the combinations thereof. The 17 ablated grammars are then evaluated via targeted syntactic evaluation on the SyntaxGym benchmark. The results of our syntactic ablation demonstrated that (i) the RNNG with zero syntactic supervision underperformed the RNNGs with some syntactic supervision, (ii) the RNNG with full syntactic supervision underperformed the RNNGs with less syntactic supervision, and (iii) the RNNG with mild syntactic supervision achieved the best performance comparable to the state-of-the-art GPT-2-XL. Those results may suggest that the “good enough” approach to language processing seems to make LMs more human-like.- Anthology ID:
- 2023.findings-eacl.173
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2300–2305
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.173
- DOI:
- 10.18653/v1/2023.findings-eacl.173
- Cite (ACL):
- Hiroshi Noji and Yohei Oseki. 2023. How Much Syntactic Supervision is “Good Enough”?. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2300–2305, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- How Much Syntactic Supervision is “Good Enough”? (Noji & Oseki, Findings 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.findings-eacl.173.pdf