Abstract
Zero-shot event extraction (EE) methods infer richly structured event records from text, based only on a minimal user specification and no training examples, which enables flexibility in exploring and developing applications. Most event extraction research uses the Automatic Content Extraction (ACE) annotated dataset to evaluate supervised EE methods, but can it be used to evaluate zero-shot and other low-supervision EE? We describe ACE’s event structures and identify significant ambiguities and issues in current evaluation practice, including (1) coreferent argument mentions, (2) conflicting argument head conventions, and (3) ignorance of modality and event class details. By sometimes mishandling these subtleties, current work may dramatically understate the actual performance of zero-shot and other low-supervision EE, considering up to 32% of correctly identified arguments and 25% of correctly ignored event mentions as false negatives. For each issue, we propose recommendations for future evaluations so the research community can better utilize ACE as an event evaluation resource.- Anthology ID:
- 2023.acl-short.142
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1651–1665
- Language:
- URL:
- https://aclanthology.org/2023.acl-short.142
- DOI:
- 10.18653/v1/2023.acl-short.142
- Cite (ACL):
- Erica Cai and Brendan O’Connor. 2023. Evaluating Zero-Shot Event Structures: Recommendations for Automatic Content Extraction (ACE) Annotations. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1651–1665, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Evaluating Zero-Shot Event Structures: Recommendations for Automatic Content Extraction (ACE) Annotations (Cai & O’Connor, ACL 2023)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/2023.acl-short.142.pdf