Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, Deepak Ramachandran
Abstract
Many Question-Answering (QA) datasets contain unanswerable questions, but their treatment in QA systems remains primitive. Our analysis of the Natural Questions (Kwiatkowski et al. 2019) dataset reveals that a substantial portion of unanswerable questions (~21%) can be explained based on the presence of unverifiable presuppositions. Through a user preference study, we demonstrate that the oracle behavior of our proposed system—which provides responses based on presupposition failure—is preferred over the oracle behavior of existing QA systems. Then, we present a novel framework for implementing such a system in three steps: presupposition generation, presupposition verification, and explanation generation, reporting progress on each. Finally, we show that a simple modification of adding presuppositions and their verifiability to the input of a competitive end-to-end QA system yields modest gains in QA performance and unanswerability detection, demonstrating the promise of our approach.- Anthology ID:
- 2021.acl-long.304
- Volume:
- Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2021
- Address:
- Online
- Editors:
- Chengqing Zong, Fei Xia, Wenjie Li, Roberto Navigli
- Venues:
- ACL | IJCNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3932–3945
- Language:
- URL:
- https://aclanthology.org/2021.acl-long.304
- DOI:
- 10.18653/v1/2021.acl-long.304
- Cite (ACL):
- Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3932–3945, Online. Association for Computational Linguistics.
- Cite (Informal):
- Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering (Kim et al., ACL-IJCNLP 2021)
- PDF:
- https://preview.aclanthology.org/landing_page/2021.acl-long.304.pdf
- Data
- FEVER, GLUE, MultiNLI, Natural Questions, QNLI