Abstract
Children acquiring English make systematic errors on subject control sentences even after they have reached near-adult competence (Chomsky, 1969), possibly due to heuristics based on semantic roles (Maratsos, 1974).Given the advanced fluency of large generative language models, we ask whether model outputs are consistent with these heuristics, and to what degree different models are consistent with each other. We find that models can be categorized by behavior into three separate groups, with broad differences between the groups. The outputs of models in the largest group are consistent with positional heuristics that succeed on subject control but fail on object control. This result is surprising, given that object control is orders of magnitude more frequent in the text data used to train such models. We examine to what degree the models are sensitive to prompting with agent-patient information, finding that raising the salience of agent and patient relations results in significant changes in the outputs of most models. Based on this observation, we leverage an existing dataset of semantic proto-role annotations (White et al. 2020) to explore the connections between control and labeling event participants with properties typically associated with agents and patients.- Anthology ID:
- 2022.emnlp-main.760
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 11065–11076
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.760
- DOI:
- 10.18653/v1/2022.emnlp-main.760
- Cite (ACL):
- Elias Stengel-Eskin and Benjamin Van Durme. 2022. The Curious Case of Control. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11065–11076, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- The Curious Case of Control (Stengel-Eskin & Van Durme, EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.emnlp-main.760.pdf