Abstract
Retrieval-Augmented Language Models (RALMs) have significantly improved performance in open-domain question answering (QA) by leveraging external knowledge. However, RALMs still struggle with unanswerable queries, where the retrieved contexts do not contain the correct answer, and with conflicting information, where different sources provide contradictory answers due to imperfect retrieval. This study introduces an in-context learning-based approach to enhance the reasoning capabilities of RALMs, making them more robust in imperfect retrieval scenarios. Our method incorporates Machine Reading Comprehension (MRC) demonstrations, referred to as cases, to boost the model’s capabilities to identify unanswerabilities and conflicts among the retrieved contexts. Experiments on two open-domain QA datasets show that our approach increases accuracy in identifying unanswerable and conflicting scenarios without requiring additional fine-tuning. This work demonstrates that in-context learning can effectively enhance the robustness of RALMs in open-domain QA tasks.- Anthology ID:
- 2024.knowledgenlp-1.7
- Volume:
- Proceedings of the 3rd Workshop on Knowledge Augmented Methods for NLP
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Wenhao Yu, Weijia Shi, Michihiro Yasunaga, Meng Jiang, Chenguang Zhu, Hannaneh Hajishirzi, Luke Zettlemoyer, Zhihan Zhang
- Venues:
- KnowledgeNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 93–102
- Language:
- URL:
- https://aclanthology.org/2024.knowledgenlp-1.7
- DOI:
- Cite (ACL):
- SeongIl Park, Seungwoo Choi, Nahyun Kim, and Jay-Yoon Lee. 2024. Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning. In Proceedings of the 3rd Workshop on Knowledge Augmented Methods for NLP, pages 93–102, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Enhancing Robustness of Retrieval-Augmented Language Models with In-Context Learning (Park et al., KnowledgeNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.knowledgenlp-1.7.pdf