Knowledge Corpus Error in Question Answering

Yejoon Lee, Philhoon Oh, James Thorne


Abstract
Recent works in open-domain question answering (QA) have explored generating context passages from large language models (LLMs), replacing the traditional retrieval step in the QA pipeline. However, it is not well understood why generated passages can be more effective than retrieved ones. This study revisits the conventional formulation of QA and introduces the concept of knowledge corpus error. This error arises when the knowledge corpus used for retrieval is only a subset of the entire string space, potentially excluding more helpful passages that exist outside the corpus. LLMs may mitigate this shortcoming by generating passages in a larger space. We come up with an experiment of paraphrasing human-annotated gold context using LLMs to observe knowledge corpus error empirically. Our results across three QA benchmarks reveal an increased performance (10% - 13%) when using paraphrased passage, indicating a signal for the existence of knowledge corpus error.
Anthology ID:
2023.findings-emnlp.616
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
9183–9197
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.616
DOI:
10.18653/v1/2023.findings-emnlp.616
Bibkey:
Cite (ACL):
Yejoon Lee, Philhoon Oh, and James Thorne. 2023. Knowledge Corpus Error in Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9183–9197, Singapore. Association for Computational Linguistics.
Cite (Informal):
Knowledge Corpus Error in Question Answering (Lee et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2023.findings-emnlp.616.pdf