Abstract
We present ReasonBert, a pre-training method that augments language models with the ability to reason over long-range relations and multiple, possibly hybrid contexts. Unlike existing pre-training methods that only harvest learning signals from local contexts of naturally occurring texts, we propose a generalized notion of distant supervision to automatically connect multiple pieces of text and tables to create pre-training examples that require long-range reasoning. Different types of reasoning are simulated, including intersecting multiple pieces of evidence, bridging from one piece of evidence to another, and detecting unanswerable cases. We conduct a comprehensive evaluation on a variety of extractive question answering datasets ranging from single-hop to multi-hop and from text-only to table-only to hybrid that require various reasoning capabilities and show that ReasonBert achieves remarkable improvement over an array of strong baselines. Few-shot experiments further demonstrate that our pre-training method substantially improves sample efficiency.- Anthology ID:
- 2021.emnlp-main.494
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6112–6127
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.494
- DOI:
- 10.18653/v1/2021.emnlp-main.494
- Cite (ACL):
- Xiang Deng, Yu Su, Alyssa Lees, You Wu, Cong Yu, and Huan Sun. 2021. ReasonBERT: Pre-trained to Reason with Distant Supervision. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6112–6127, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- ReasonBERT: Pre-trained to Reason with Distant Supervision (Deng et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2021.emnlp-main.494.pdf
- Code
- sunlab-osu/reasonbert
- Data
- GraphQuestions, HotpotQA, HybridQA, MRQA, Natural Questions, NewsQA, SQuAD, SearchQA, TriviaQA