NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset

Qiyuan Zhang, Lei Wang, Sicheng Yu, Shuohang Wang, Yang Wang, Jing Jiang, Ee-Peng Lim


Abstract
While diverse question answering (QA) datasets have been proposed and contributed significantly to the development of deep learning models for QA tasks, the existing datasets fall short in two aspects. First, we lack QA datasets covering complex questions that involve answers as well as the reasoning processes to get them. As a result, the state-of-the-art QA research on numerical reasoning still focuses on simple calculations and does not provide the mathematical expressions or evidence justifying the answers. Second, the QA community has contributed a lot of effort to improve the interpretability of QA models. However, they fail to explicitly show the reasoning process, such as the evidence order for reasoning and the interactions between different pieces of evidence. To address the above shortcoming, we introduce NOAHQA, a conversational and bilingual QA dataset with questions requiring numerical reasoning with compound mathematical expressions. With NOAHQA, we develop an interpretable reasoning graph as well as the appropriate evaluation metric to measure the answer quality. We evaluate the state-of-the-art QA models trained using existing QA datasets on NOAHQA and show that the best among them can only achieve 55.5 exact match scores, while the human performance is 89.7. We also present a new QA model for generating a reasoning graph where the reasoning graph metric still has a large gap compared with that of humans, eg, 28 scores.
Anthology ID:
2021.findings-emnlp.350
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
4147–4161
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.350
DOI:
10.18653/v1/2021.findings-emnlp.350
Bibkey:
Cite (ACL):
Qiyuan Zhang, Lei Wang, Sicheng Yu, Shuohang Wang, Yang Wang, Jing Jiang, and Ee-Peng Lim. 2021. NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4147–4161, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
NOAHQA: Numerical Reasoning with Interpretable Graph Question Answering Dataset (Zhang et al., Findings 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.findings-emnlp.350.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2021.findings-emnlp.350.mp4
Code
 don-joey/noahqa
Data
CoQADROPHotpotQAMAWPSMath23K