Collective Relevance Labeling for Passage Retrieval

Jihyuk Kim, Minsoo Kim, Seung-won Hwang


Abstract
Deep learning for Information Retrieval (IR) requires a large amount of high-quality query-document relevance labels, but such labels are inherently sparse. Label smoothing redistributes some observed probability mass over unobserved instances, often uniformly, uninformed of the true distribution. In contrast, we propose knowledge distillation for informed labeling, without incurring high computation overheads at evaluation time. Our contribution is designing a simple but efficient teacher model which utilizes collective knowledge, to outperform state-of-the-arts distilled from a more complex teacher model. Specifically, we train up to ×8 faster than the state-of-the-art teacher, while distilling the rankings better. Our code is publicly available at https://github.com/jihyukkim-nlp/CollectiveKD.
Anthology ID:
2022.naacl-main.305
Volume:
Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Month:
July
Year:
2022
Address:
Seattle, United States
Venue:
NAACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
4141–4147
Language:
URL:
https://aclanthology.org/2022.naacl-main.305
DOI:
10.18653/v1/2022.naacl-main.305
Bibkey:
Cite (ACL):
Jihyuk Kim, Minsoo Kim, and Seung-won Hwang. 2022. Collective Relevance Labeling for Passage Retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4141–4147, Seattle, United States. Association for Computational Linguistics.
Cite (Informal):
Collective Relevance Labeling for Passage Retrieval (Kim et al., NAACL 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.naacl-main.305.pdf
Software:
 2022.naacl-main.305.software.tgz
Video:
 https://preview.aclanthology.org/ingestion-script-update/2022.naacl-main.305.mp4
Code
 jihyukkim-nlp/CollectiveKD
Data
MS MARCO