Reasoning about Goals, Steps, and Temporal Ordering with WikiHow

Li Zhang, Qing Lyu, Chris Callison-Burch


Abstract
We propose a suite of reasoning tasks on two types of relations between procedural events: goal-step relations (“learn poses” is a step in the larger goal of “doing yoga”) and step-step temporal relations (“buy a yoga mat” typically precedes “learn poses”). We introduce a dataset targeting these two relations based on wikiHow, a website of instructional how-to articles. Our human-validated test set serves as a reliable benchmark for common-sense inference, with a gap of about 10% to 20% between the performance of state-of-the-art transformer models and human performance. Our automatically-generated training set allows models to effectively transfer to out-of-domain tasks requiring knowledge of procedural events, with greatly improved performances on SWAG, Snips, and Story Cloze Test in zero- and few-shot settings.
Anthology ID:
2020.emnlp-main.374
Original:
2020.emnlp-main.374v1
Version 2:
2020.emnlp-main.374v2
Version 3:
2020.emnlp-main.374v3
Volume:
Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
Month:
November
Year:
2020
Address:
Online
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4630–4639
Language:
URL:
https://aclanthology.org/2020.emnlp-main.374
DOI:
10.18653/v1/2020.emnlp-main.374
Bibkey:
Cite (ACL):
Li Zhang, Qing Lyu, and Chris Callison-Burch. 2020. Reasoning about Goals, Steps, and Temporal Ordering with WikiHow. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4630–4639, Online. Association for Computational Linguistics.
Cite (Informal):
Reasoning about Goals, Steps, and Temporal Ordering with WikiHow (Zhang et al., EMNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.emnlp-main.374.pdf
Optional supplementary material:
 2020.emnlp-main.374.OptionalSupplementaryMaterial.zip
Video:
 https://slideslive.com/38938737
Code
 zharry29/wikihow-goal-step
Data
ROCStoriesSNIPSSWAGWikiHow