Robust Machine Reading Comprehension by Learning Soft labels
Zhenyu Zhao, Shuangzhi Wu, Muyun Yang, Kehai Chen, Tiejun Zhao
Abstract
Neural models have achieved great success on the task of machine reading comprehension (MRC), which are typically trained on hard labels. We argue that hard labels limit the model capability on generalization due to the label sparseness problem. In this paper, we propose a robust training method for MRC models to address this problem. Our method consists of three strategies, 1) label smoothing, 2) word overlapping, 3) distribution prediction. All of them help to train models on soft labels. We validate our approach on the representative architecture - ALBERT. Experimental results show that our method can greatly boost the baseline with 1% improvement in average, and achieve state-of-the-art performance on NewsQA and QUOREF.- Anthology ID:
- 2020.coling-main.248
- Volume:
- Proceedings of the 28th International Conference on Computational Linguistics
- Month:
- December
- Year:
- 2020
- Address:
- Barcelona, Spain (Online)
- Venue:
- COLING
- SIG:
- Publisher:
- International Committee on Computational Linguistics
- Note:
- Pages:
- 2754–2759
- Language:
- URL:
- https://aclanthology.org/2020.coling-main.248
- DOI:
- 10.18653/v1/2020.coling-main.248
- Cite (ACL):
- Zhenyu Zhao, Shuangzhi Wu, Muyun Yang, Kehai Chen, and Tiejun Zhao. 2020. Robust Machine Reading Comprehension by Learning Soft labels. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2754–2759, Barcelona, Spain (Online). International Committee on Computational Linguistics.
- Cite (Informal):
- Robust Machine Reading Comprehension by Learning Soft labels (Zhao et al., COLING 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.coling-main.248.pdf
- Data
- NewsQA, Quoref