MTR: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning
Yitian Li, Jidong Tian, Caoyun Fan, Wenqing Chen, Hao He, Yaohui Jin
Abstract
A long-standing difficulty in AI is the introduction of human-like reasoning in machine reading comprehension. Since algorithmic models can already perform as well as humans on simple quality assurance tasks thanks to the development of deep learning techniques, more difficult reasoning datasets have been presented. However, these datasets mainly focus on a single type of reasoning. There are still significant gaps in the studies when compared to the complex reasoning used in daily life. In this work, we introduce a brand-new dataset, named MTR. There are two parts to it: the first combines deductive and inductive reasoning, and the second does the same with inductive and defeasible reasoning. It consists of more than 30k QA instances, inferring relations between characters in short stories. Results show that state-of-the-art neural models do noticeably worse than expected. Our empirical results highlight the gap in the models’ ability to handle sophisticated inference.- Anthology ID:
- 2023.findings-acl.640
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10078–10089
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.640
- DOI:
- 10.18653/v1/2023.findings-acl.640
- Cite (ACL):
- Yitian Li, Jidong Tian, Caoyun Fan, Wenqing Chen, Hao He, and Yaohui Jin. 2023. MTR: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10078–10089, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- MTR: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning (Li et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.findings-acl.640.pdf