Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval
Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten Rijke, Yixing Fan, Xueqi Cheng
Abstract
Generative retrieval uses differentiable search indexes to directly generate relevant document identifiers in response to a query. Recent studies have highlighted the potential of a strong generative retrieval model, trained with carefully crafted pre-training tasks, to enhance downstream retrieval tasks via fine-tuning. However, the full power of pre-training for generative retrieval remains underexploited due to its reliance on pre-defined static document identifiers, which may not align with evolving model parameters. In this work, we introduce BootRet, a bootstrapped pre-training method for generative retrieval that dynamically adjusts document identifiers during pre-training to accommodate the continuing memorization of the corpus. BootRet involves three key training phases: (i) initial identifier generation, (ii) pre-training via corpus indexing and relevance prediction tasks, and (iii) bootstrapping for identifier updates. To facilitate the pre-training phase, we further introduce noisy documents and pseudo-queries, generated by large language models, to resemble semantic connections in both indexing and retrieval tasks. Experimental results demonstrate that BootRet significantly outperforms existing pre-training generative retrieval baselines and performs well even in zero-shot settings.- Anthology ID:
- 2024.findings-acl.614
- Volume:
- Findings of the Association for Computational Linguistics ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand and virtual meeting
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 10303–10317
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.614
- DOI:
- 10.18653/v1/2024.findings-acl.614
- Cite (ACL):
- Yubao Tang, Ruqing Zhang, Jiafeng Guo, Maarten Rijke, Yixing Fan, and Xueqi Cheng. 2024. Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval. In Findings of the Association for Computational Linguistics ACL 2024, pages 10303–10317, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
- Cite (Informal):
- Bootstrapped Pre-training with Dynamic Identifier Prediction for Generative Retrieval (Tang et al., Findings 2024)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/2024.findings-acl.614.pdf