Abstract
Levin et al. (2019) show experimentally that the interpretations of novel English noun compounds (e.g., stew skillet), while not fully compositional, are highly predictable based on whether the modifier and head refer to artifacts or natural kinds. Is the large language model GPT-3 governed by the same interpretive principles? To address this question, we first compare Levin et al.’s experimental data with GPT-3 generations, finding a high degree of similarity. However, this evidence is consistent with GPT-3 reasoning only about specific lexical items rather than the more abstract conceptual categories of Levin et al.’s theory. To probe more deeply, we construct prompts that require the relevant kind of conceptual reasoning. Here, we fail to find convincing evidence that GPT-3 is reasoning about more than just individual lexical items. These results highlight the importance of controlling for low-level distributional regularities when assessing whether a large language model latently encodes a deeper theory.- Anthology ID:
- 2022.findings-emnlp.50
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 717–728
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.50
- DOI:
- 10.18653/v1/2022.findings-emnlp.50
- Cite (ACL):
- Siyan Li, Riley Carlson, and Christopher Potts. 2022. Systematicity in GPT-3’s Interpretation of Novel English Noun Compounds. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 717–728, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Systematicity in GPT-3’s Interpretation of Novel English Noun Compounds (Li et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/naacl-24-ws-corrections/2022.findings-emnlp.50.pdf