Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix

Kuan-Hao Huang, Liang Tan, Rui Hou, Sinong Wang, Amjad Almahairi, Ruty Rinott

Abstract
Many real-world applications require making multiple predictions from the same text. Fine-tuning a large pre-trained language model for each downstream task causes computational burdens in the inference time due to several times of forward passes. To amortize the computational cost, freezing the language model and building lightweight models for downstream tasks based on fixed text representations are common solutions. Accordingly, how to learn fixed but general text representations that can generalize well to unseen downstream tasks becomes a challenge. Previous works have shown that the generalizability of representations can be improved by fine-tuning the pre-trained language model with some source tasks in a multi-tasking way. In this work, we propose a prefix-based method to learn the fixed text representations with source tasks. We learn a task-specific prefix for each source task independently and combine them to get the final representations. Our experimental results show that prefix-based training performs better than multi-tasking training and can update the text representations at a smaller computational cost than multi-tasking training.
Anthology ID:
2023.findings-emnlp.497
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
7422–7430
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.497
DOI:
10.18653/v1/2023.findings-emnlp.497
Bibkey:
Cite (ACL):
Kuan-Hao Huang, Liang Tan, Rui Hou, Sinong Wang, Amjad Almahairi, and Ruty Rinott. 2023. Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7422–7430, Singapore. Association for Computational Linguistics.
Cite (Informal):
Learning Easily Updated General Purpose Text Representations with Adaptable Task-Specific Prefix (Huang et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2023.findings-emnlp.497.pdf
PDF Search