Evidence Sentence Extraction for Machine Reading Comprehension

Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, Dan Roth

Abstract
Remarkable success has been achieved in the last few years on some limited machine reading comprehension (MRC) tasks. However, it is still difficult to interpret the predictions of existing MRC models. In this paper, we focus on extracting evidence sentences that can explain or support the answers of multiple-choice MRC tasks, where the majority of answer options cannot be directly extracted from reference documents. Due to the lack of ground truth evidence sentence labels in most cases, we apply distant supervision to generate imperfect labels and then use them to train an evidence sentence extractor. To denoise the noisy labels, we apply a recently proposed deep probabilistic logic learning framework to incorporate both sentence-level and cross-sentence linguistic indicators for indirect supervision. We feed the extracted evidence sentences into existing MRC models and evaluate the end-to-end performance on three challenging multiple-choice MRC datasets: MultiRC, RACE, and DREAM, achieving comparable or better performance than the same models that take as input the full reference document. To the best of our knowledge, this is the first work extracting evidence sentences for multiple-choice MRC.
Anthology ID:
K19-1065
Volume:
Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Mohit Bansal, Aline Villavicencio
Venue:
CoNLL
SIG:
SIGNLL
Publisher:
Association for Computational Linguistics
Note:
Pages:
696–707
Language:
URL:
https://preview.aclanthology.org/build-pipeline-with-new-library/K19-1065/
DOI:
10.18653/v1/K19-1065
Bibkey:
Cite (ACL):
Hai Wang, Dian Yu, Kai Sun, Jianshu Chen, Dong Yu, David McAllester, and Dan Roth. 2019. Evidence Sentence Extraction for Machine Reading Comprehension. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 696–707, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Evidence Sentence Extraction for Machine Reading Comprehension (Wang et al., CoNLL 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/build-pipeline-with-new-library/K19-1065.pdf
Supplementary material:
 K19-1065.Supplementary_Material.pdf
Attachment:
 K19-1065.Attachment.pdf
Code
 nlpdata/evidence
Data
MultiRC, RACE
PDF Search Code Supplementary material Attachment Fix metadata