On Event Individuation for Document-Level Information Extraction

William Gantt, Reno Kriz, Yunmo Chen, Siddharth Vashishtha, Aaron White

Abstract
As information extraction (IE) systems have grown more adept at processing whole documents, the classic task of *template filling* has seen renewed interest as a benchmark for document-level IE. In this position paper, we call into question the suitability of template filling for this purpose. We argue that the task demands definitive answers to thorny questions of *event individuation* — the problem of distinguishing distinct events — about which even human experts disagree. Through an annotation study and error analysis, we show that this raises concerns about the usefulness of template filling metrics, the quality of datasets for the task, and the ability of models to learn it. Finally, we consider possible solutions.
Anthology ID:
2023.findings-emnlp.862
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12938–12958
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.862
DOI:
10.18653/v1/2023.findings-emnlp.862
Bibkey:
Cite (ACL):
William Gantt, Reno Kriz, Yunmo Chen, Siddharth Vashishtha, and Aaron White. 2023. On Event Individuation for Document-Level Information Extraction. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12938–12958, Singapore. Association for Computational Linguistics.
Cite (Informal):
On Event Individuation for Document-Level Information Extraction (Gantt et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/2023.findings-emnlp.862.pdf
PDF Search