Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval

Fan Jiang, Qiongkai Xu, Tom Drummond, Trevor Cohn

Abstract
Neural ‘dense’ retrieval models are state of the art for many datasets, however these models often exhibit limited domain transfer ability. Existing approaches to adaptation are unwieldy, such as requiring explicit supervision, complex model architectures, or massive external models. We present ABEL, a simple but effective unsupervised method to enhance passage retrieval in zero-shot settings. Our technique follows a straightforward loop: a dense retriever learns from supervision signals provided by a reranker, and subsequently, the reranker is updated based on feedback from the improved retriever. By iterating this loop, the two components mutually enhance one another’s performance. Experimental results demonstrate that our unsupervised ABEL model outperforms both leading supervised and unsupervised retrievers on the BEIR benchmark. Meanwhile, it exhibits strong adaptation abilities to tasks and domains that were unseen during training. By either fine-tuning ABEL on labelled data or integrating it with existing supervised dense retrievers, we achieve state-of-the-art results.
Anthology ID:
2023.findings-emnlp.65
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
912–931
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.65
DOI:
10.18653/v1/2023.findings-emnlp.65
Bibkey:
Cite (ACL):
Fan Jiang, Qiongkai Xu, Tom Drummond, and Trevor Cohn. 2023. Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 912–931, Singapore. Association for Computational Linguistics.
Cite (Informal):
Boot and Switch: Alternating Distillation for Zero-Shot Dense Retrieval (Jiang et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/2023.findings-emnlp.65.pdf
PDF Search