Abstract
We propose a novel challenge for large language models: ellipsis-dependent reasoning. We define several structures of paired examples, where an ellipsis example is matched to its non-ellipsis counterpart, and a question is posed which requires resolution of the ellipsis. Test results show that the best models perform well on non-elliptical examples but struggle with all but the simplest ellipsis structures.- Anthology ID:
- 2023.acl-short.4
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 39–47
- Language:
- URL:
- https://aclanthology.org/2023.acl-short.4
- DOI:
- 10.18653/v1/2023.acl-short.4
- Cite (ACL):
- Daniel Hardt. 2023. Ellipsis-Dependent Reasoning: a New Challenge for Large Language Models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 39–47, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Ellipsis-Dependent Reasoning: a New Challenge for Large Language Models (Hardt, ACL 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.acl-short.4.pdf