Improving Multitask Retrieval by Promoting Task Specialization
Wenzheng Zhang, Chenyan Xiong, Karl Stratos, Arnold Overwijk
Abstract
In multitask retrieval, a single retriever is trained to retrieve relevant contexts for multiple tasks. Despite its practical appeal, naive multitask retrieval lags behind task-specific retrieval, in which a separate retriever is trained for each task. We show that it is possible to train a multitask retriever that outperforms task-specific retrievers by promoting task specialization. The main ingredients are: (1) a better choice of pretrained model—one that is explicitly optimized for multitasking—along with compatible prompting, and (2) a novel adaptive learning method that encourages each parameter to specialize in a particular task. The resulting multitask retriever is highly performant on the KILT benchmark. Upon analysis, we find that the model indeed learns parameters that are more task-specialized compared to naive multitasking without prompting or adaptive learning.1- Anthology ID:
- 2023.tacl-1.68
- Volume:
- Transactions of the Association for Computational Linguistics, Volume 11
- Month:
- Year:
- 2023
- Address:
- Cambridge, MA
- Venue:
- TACL
- SIG:
- Publisher:
- MIT Press
- Note:
- Pages:
- 1201–1212
- Language:
- URL:
- https://aclanthology.org/2023.tacl-1.68
- DOI:
- 10.1162/tacl_a_00597
- Cite (ACL):
- Wenzheng Zhang, Chenyan Xiong, Karl Stratos, and Arnold Overwijk. 2023. Improving Multitask Retrieval by Promoting Task Specialization. Transactions of the Association for Computational Linguistics, 11:1201–1212.
- Cite (Informal):
- Improving Multitask Retrieval by Promoting Task Specialization (Zhang et al., TACL 2023)
- PDF:
- https://preview.aclanthology.org/ml4al-ingestion/2023.tacl-1.68.pdf