Abstract
The Story Cloze Test (SCT) is designed for training and evaluating machine learning algorithms for narrative understanding and inferences. The SOTA models can achieve over 90% accuracy on predicting the last sentence. However, it has been shown that high accuracy can be achieved by merely using surface-level features. We suspect these models may not truly understand the story. Based on the SCT dataset, we constructed a human-labeled and human-verified commonsense knowledge inference dataset. Given the first four sentences of a story, we asked crowd-source workers to choose from four types of narrative inference for deciding the ending sentence and which sentence contributes most to the inference. We accumulated data on 1871 stories, and three human workers labeled each story. Analysis of the intra-category and inter-category agreements show a high level of consensus. We present two new tasks for predicting the narrative inference categories and contributing sentences. Our results show that transformer-based models can reach SOTA performance on the original SCT task using transfer learning but don’t perform well on these new and more challenging tasks.- Anthology ID:
- 2022.lrec-1.375
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 3500–3508
- Language:
- URL:
- https://aclanthology.org/2022.lrec-1.375
- DOI:
- Cite (ACL):
- Bingsheng Yao, Ethan Joseph, Julian Lioanag, and Mei Si. 2022. A Corpus for Commonsense Inference in Story Cloze Test. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3500–3508, Marseille, France. European Language Resources Association.
- Cite (Informal):
- A Corpus for Commonsense Inference in Story Cloze Test (Yao et al., LREC 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2022.lrec-1.375.pdf
- Data
- StoryCloze