Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes
Haoyu Wang, Hongming Zhang, Yueguan Wang, Yuqian Deng, Muhao Chen, Dan Roth
Abstract
Natural language often describes events in different granularities, such that more coarse-grained (goal) events can often be decomposed into fine-grained sequences of (step) events. A critical but overlooked challenge in understanding an event process lies in the fact that the step events are not equally important to the central goal. In this paper, we seek to fill this gap by studying how well current models can understand the essentiality of different step events towards a goal event. As discussed by cognitive studies, such an ability enables the machine to mimic human’s commonsense reasoning about preconditions and necessary efforts of daily-life tasks. Our work contributes with a high-quality corpus of (goal, step) pairs from a community guideline website WikiHow, where the steps are manually annotated with their essentiality w.r.t. the goal. The high IAA indicates that humans have a consistent understanding of the events. Despite evaluating various statistical and massive pre-trained NLU models, we observe that existing SOTA models all perform drastically behind humans, indicating the need for future investigation of this crucial yet challenging task.- Anthology ID:
- 2023.emnlp-main.246
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4048–4056
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.246
- DOI:
- 10.18653/v1/2023.emnlp-main.246
- Cite (ACL):
- Haoyu Wang, Hongming Zhang, Yueguan Wang, Yuqian Deng, Muhao Chen, and Dan Roth. 2023. Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4048–4056, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Are All Steps Equally Important? Benchmarking Essentiality Detection in Event Processes (Wang et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2023.emnlp-main.246.pdf