Abstract
Retriever-reader models achieve competitive performance across many different NLP tasks such as open question answering and dialogue conversations. In this work, we notice these models easily overfit the top-rank retrieval passages and standard training fails to reason over the entire retrieval passages. We introduce a learnable passage mask mechanism which desensitizes the impact from the top-rank retrieval passages and prevents the model from overfitting. Controlling the gradient variance with fewer mask candidates and selecting the mask candidates with one-shot bi-level optimization, our learnable regularization strategy enforces the answer generation to focus on the entire retrieval passages. Experiments on different tasks across open question answering, dialogue conversation, and fact verification show that our method consistently outperforms its baselines. Extensive experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.- Anthology ID:
- 2022.emnlp-main.260
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3931–3943
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.260
- DOI:
- 10.18653/v1/2022.emnlp-main.260
- Cite (ACL):
- Shujian Zhang, Chengyue Gong, and Xingchao Liu. 2022. Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3931–3943, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Passage-Mask: A Learnable Regularization Strategy for Retriever-Reader Models (Zhang et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2022.emnlp-main.260.pdf