Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond

Xinyu Wang, Hainiu Xu, Lin Gui, Yulan He

Abstract
Task embedding, a meta-learning technique that captures task-specific information, has gained popularity, especially in areas such as multi-task learning, model editing, and interpretability. However, it faces challenges with the emergence of prompt-guided Large Language Models (LLMs) operating in a gradient-free manner. Existing task embedding methods rely on fine-tuned, task-specific language models, which hinders the adaptability of task embeddings across diverse models, especially prompt-based LLMs. To hardness the potential of task embeddings in the era of LLMs, we propose a framework for unified task embeddings (FUTE), harmonizing task embeddings from various models, including smaller language models and LLMs with varied prompts, within a single vector space. Such uniformity enables comparison and analysis of similarities amongst different models, broadening the scope and utility of existing task embedding methods in multi-model scenarios, while maintaining their performance comparable to architecture-specific methods.
Anthology ID:
2024.findings-acl.493
Volume:
Findings of the Association for Computational Linguistics: ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
8324–8340
Language:
URL:
https://preview.aclanthology.org/add_missing_videos/2024.findings-acl.493/
DOI:
10.18653/v1/2024.findings-acl.493
Bibkey:
Cite (ACL):
Xinyu Wang, Hainiu Xu, Lin Gui, and Yulan He. 2024. Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond. In Findings of the Association for Computational Linguistics: ACL 2024, pages 8324–8340, Bangkok, Thailand. Association for Computational Linguistics.
Cite (Informal):
Towards Unified Task Embeddings Across Multiple Models: Bridging the Gap for Prompt-Based Large Language Models and Beyond (Wang et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/add_missing_videos/2024.findings-acl.493.pdf
PDF Search Fix data