GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks
Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, Soroush Vosoughi
Abstract
A key problem in multi-task learning (MTL) research is how to select high-quality auxiliary tasks automatically. This paper presents GradTS, an automatic auxiliary task selection method based on gradient calculation in Transformer-based models. Compared to AUTOSEM, a strong baseline method, GradTS improves the performance of MT-DNN with a bert-base-cased backend model, from 0.33% to 17.93% on 8 natural language understanding (NLU) tasks in the GLUE benchmarks. GradTS is also time-saving since (1) its gradient calculations are based on single-task experiments and (2) the gradients are re-used without additional experiments when the candidate task set changes. On the 8 GLUE classification tasks, for example, GradTS costs on average 21.32% less time than AUTOSEM with comparable GPU consumption. Further, we show the robustness of GradTS across various task settings and model selections, e.g. mixed objectives among candidate tasks. The efficiency and efficacy of GradTS in these case studies illustrate its general applicability in MTL research without requiring manual task filtering or costly parameter tuning.- Anthology ID:
- 2021.emnlp-main.455
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5621–5632
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.455
- DOI:
- 10.18653/v1/2021.emnlp-main.455
- Cite (ACL):
- Weicheng Ma, Renze Lou, Kai Zhang, Lili Wang, and Soroush Vosoughi. 2021. GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5621–5632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks (Ma et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/ingest-2024-clasp/2021.emnlp-main.455.pdf
- Data
- GLUE, MELD, QNLI