CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking

George Zerveas, Navid Rekabsaz, Daniel Cohen, Carsten Eickhoff


Abstract
Contrastive learning has been the dominant approach to training dense retrieval models. In this work, we investigate the impact of ranking context - an often overlooked aspect of learning dense retrieval models. In particular, we examine the effect of its constituent parts: jointly scoring a large number of negatives per query, using retrieved (query-specific) instead of random negatives, and a fully list-wise loss. To incorporate these factors into training, we introduce Contextual Document Embedding Reranking (CODER), a highly efficient retrieval framework. When reranking, it incurs only a negligible computational overhead on top of a first-stage method at run time (approx. 5 ms delay per query), allowing it to be easily combined with any state-of-the-art dual encoder method. Models trained through CODER can also be used as stand-alone retrievers. Evaluating CODER in a large set of experiments on the MS MARCO and TripClick collections, we show that the contextual reranking of precomputed document embeddings leads to a significant improvement in retrieval performance. This improvement becomes even more pronounced when more relevance information per query is available, shown in the TripClick collection, where we establish new state-of-the-art results by a large margin.
Anthology ID:
2022.emnlp-main.727
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
10626–10644
Language:
URL:
https://aclanthology.org/2022.emnlp-main.727
DOI:
10.18653/v1/2022.emnlp-main.727
Bibkey:
Cite (ACL):
George Zerveas, Navid Rekabsaz, Daniel Cohen, and Carsten Eickhoff. 2022. CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10626–10644, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
CODER: An efficient framework for improving retrieval through COntextual Document Embedding Reranking (Zerveas et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl-2023-videos/2022.emnlp-main.727.pdf