SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, Roger Levy
Abstract
Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale language models. We present SyntaxGym, an online platform designed to make targeted evaluations accessible to both experts in NLP and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community: 1. A website, syntaxgym.org, which centralizes the process of targeted syntactic evaluation and provides easy tools for analysis and visualization; 2. Two command-line tools, ‘syntaxgym‘ and ‘lm-zoo‘, which allow any user to reproduce targeted syntactic evaluations and general language model inference on their own machine.- Anthology ID:
- 2020.acl-demos.10
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Asli Celikyilmaz, Tsung-Hsien Wen
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 70–76
- Language:
- URL:
- https://aclanthology.org/2020.acl-demos.10
- DOI:
- 10.18653/v1/2020.acl-demos.10
- Cite (ACL):
- Jon Gauthier, Jennifer Hu, Ethan Wilcox, Peng Qian, and Roger Levy. 2020. SyntaxGym: An Online Platform for Targeted Evaluation of Language Models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 70–76, Online. Association for Computational Linguistics.
- Cite (Informal):
- SyntaxGym: An Online Platform for Targeted Evaluation of Language Models (Gauthier et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2020.acl-demos.10.pdf