Improving QA Generalization by Concurrent Modeling of Multiple Biases
Mingzhu Wu, Nafise Sadat Moosavi, Andreas Rücklé, Iryna Gurevych
Abstract
Existing NLP datasets contain various biases that models can easily exploit to achieve high performances on the corresponding evaluation sets. However, focusing on dataset-specific biases limits their ability to learn more generalizable knowledge about the task from more general data patterns. In this paper, we investigate the impact of debiasing methods for improving generalization and propose a general framework for improving the performance on both in-domain and out-of-domain datasets by concurrent modeling of multiple biases in the training data. Our framework weights each example based on the biases it contains and the strength of those biases in the training data. It then uses these weights in the training objective so that the model relies less on examples with high bias weights. We extensively evaluate our framework on extractive question answering with training data from various domains with multiple biases of different strengths. We perform the evaluations in two different settings, in which the model is trained on a single domain or multiple domains simultaneously, and show its effectiveness in both settings compared to state-of-the-art debiasing methods.- Anthology ID:
- 2020.findings-emnlp.74
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2020
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Trevor Cohn, Yulan He, Yang Liu
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 839–853
- Language:
- URL:
- https://aclanthology.org/2020.findings-emnlp.74
- DOI:
- 10.18653/v1/2020.findings-emnlp.74
- Cite (ACL):
- Mingzhu Wu, Nafise Sadat Moosavi, Andreas Rücklé, and Iryna Gurevych. 2020. Improving QA Generalization by Concurrent Modeling of Multiple Biases. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 839–853, Online. Association for Computational Linguistics.
- Cite (Informal):
- Improving QA Generalization by Concurrent Modeling of Multiple Biases (Wu et al., Findings 2020)
- PDF:
- https://preview.aclanthology.org/landing_page/2020.findings-emnlp.74.pdf
- Code
- UKPLab/qa-generalization-concurrent-debiasing
- Data
- DROP, DuoRC, HotpotQA, MRQA, Natural Questions, NewsQA, RACE, SQuAD, TriviaQA