Abstract
Coreference resolution – which is a crucial task for understanding discourse and language at large – has yet to witness widespread benefits from large language models (LLMs). Moreover, coreference resolution systems largely rely on supervised labels, which are highly expensive and difficult to annotate, thus making it ripe for prompt engineering. In this paper, we introduce a QA-based prompt-engineering method and discern generative, pre-trained LLMs’ abilities and limitations toward the task of coreference resolution. Our experiments show that GPT-2 and GPT-Neo can return valid answers, but that their capabilities to identify coreferent mentions are limited and prompt-sensitive, leading to inconsistent results.- Anthology ID:
- 2022.insights-1.10
- Volume:
- Proceedings of the Third Workshop on Insights from Negative Results in NLP
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Shabnam Tafreshi, João Sedoc, Anna Rogers, Aleksandr Drozd, Anna Rumshisky, Arjun Akula
- Venue:
- insights
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 75–81
- Language:
- URL:
- https://aclanthology.org/2022.insights-1.10
- DOI:
- 10.18653/v1/2022.insights-1.10
- Cite (ACL):
- Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Chris Tanner. 2022. What GPT Knows About Who is Who. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, pages 75–81, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- What GPT Knows About Who is Who (Yang et al., insights 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-3/2022.insights-1.10.pdf
- Code
- awesomecoref/prompt-coref
- Data
- WSC