@inproceedings{kong-etal-2020-scde,
    title = "{SCDE}: Sentence Cloze Dataset with High Quality Distractors From Examinations",
    author = "Kong, Xiang  and
      Gangal, Varun  and
      Hovy, Eduard",
    editor = "Jurafsky, Dan  and
      Chai, Joyce  and
      Schluter, Natalie  and
      Tetreault, Joel",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2020.acl-main.502/",
    doi = "10.18653/v1/2020.acl-main.502",
    pages = "5668--5683",
    abstract = "We introduce SCDE, a dataset to evaluate the performance of computational models through sentence prediction. SCDE is a human created sentence cloze dataset, collected from public school English examinations. Our task requires a model to fill up multiple blanks in a passage from a shared candidate set with distractors designed by English teachers. Experimental results demonstrate that this task requires the use of non-local, discourse-level context beyond the immediate sentence neighborhood. The blanks require joint solving and significantly impair each other`s context. Furthermore, through ablations, we show that the distractors are of high quality and make the task more challenging. Our experiments show that there is a significant performance gap between advanced models (72{\%}) and humans (87{\%}), encouraging future models to bridge this gap."
}