CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation

Abhilasha Ravichander, Matt Gardner, Ana Marasovic


Abstract
The full power of human language-based communication cannot be realized without negation. All human languages have some form of negation. Despite this, negation remains a challenging phenomenon for current natural language understanding systems. To facilitate the future development of models that can process negation effectively, we present CONDAQA, the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. We collect paragraphs with diverse negation cues, then have crowdworkers ask questions about the implications of the negated statement in the passage. We also have workers make three kinds of edits to the passage—paraphrasing the negated statement, changing the scope of the negation, and reversing the negation—resulting in clusters of question-answer pairs that are difficult for models to answer with spurious shortcuts. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues and is challenging for current state-of-the-art models. The best performing model on CONDAQA (UnifiedQA-v2-3b) achieves only 42% on our consistency metric, well below human performance which is 81%. We release our dataset, along with fully-finetuned, few-shot, and zero-shot evaluations, to facilitate the development of future NLP methods that work on negated language.
Anthology ID:
2022.emnlp-main.598
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8729–8755
Language:
URL:
https://aclanthology.org/2022.emnlp-main.598
DOI:
Bibkey:
Cite (ACL):
Abhilasha Ravichander, Matt Gardner, and Ana Marasovic. 2022. CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8729–8755, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (Ravichander et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/author-url/2022.emnlp-main.598.pdf