Tackling the Story Ending Biases in The Story Cloze Test

Rishi Sharma, James Allen, Omid Bakhshandeh, Nasrin Mostafazadeh


Abstract
The Story Cloze Test (SCT) is a recent framework for evaluating story comprehension and script learning. There have been a variety of models tackling the SCT so far. Although the original goal behind the SCT was to require systems to perform deep language understanding and commonsense reasoning for successful narrative understanding, some recent models could perform significantly better than the initial baselines by leveraging human-authorship biases discovered in the SCT dataset. In order to shed some light on this issue, we have performed various data analysis and analyzed a variety of top performing models presented for this task. Given the statistics we have aggregated, we have designed a new crowdsourcing scheme that creates a new SCT dataset, which overcomes some of the biases. We benchmark a few models on the new dataset and show that the top-performing model on the original SCT dataset fails to keep up its performance. Our findings further signify the importance of benchmarking NLP systems on various evolving test sets.
Anthology ID:
P18-2119
Volume:
Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
Month:
July
Year:
2018
Address:
Melbourne, Australia
Venue:
ACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
752–757
Language:
URL:
https://aclanthology.org/P18-2119
DOI:
10.18653/v1/P18-2119
Bibkey:
Cite (ACL):
Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh. 2018. Tackling the Story Ending Biases in The Story Cloze Test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752–757, Melbourne, Australia. Association for Computational Linguistics.
Cite (Informal):
Tackling the Story Ending Biases in The Story Cloze Test (Sharma et al., ACL 2018)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/P18-2119.pdf
Presentation:
 P18-2119.Presentation.pdf
Video:
 https://vimeo.com/285806126
Data
ROCStories