Abstract
We present an efficient BERT-based multi-task (MT) framework that is particularly suitable for iterative and incremental development of the tasks. The proposed framework is based on the idea of partial fine-tuning, i.e. only fine-tune some top layers of BERT while keep the other layers frozen. For each task, we train independently a single-task (ST) model using partial fine-tuning. Then we compress the task-specific layers in each ST model using knowledge distillation. Those compressed ST models are finally merged into one MT model so that the frozen layers of the former are shared across the tasks. We exemplify our approach on eight GLUE tasks, demonstrating that it is able to achieve 99.6% of the performance of the full fine-tuning method, while reducing up to two thirds of its overhead.- Anthology ID:
- 2022.acl-short.89
- Volume:
- Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Smaranda Muresan, Preslav Nakov, Aline Villavicencio
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 785–796
- Language:
- URL:
- https://aclanthology.org/2022.acl-short.89
- DOI:
- 10.18653/v1/2022.acl-short.89
- Cite (ACL):
- Tianwen Wei, Jianwei Qi, and Shenghuan He. 2022. A Flexible Multi-Task Model for BERT Serving. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 785–796, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- A Flexible Multi-Task Model for BERT Serving (Wei et al., ACL 2022)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/2022.acl-short.89.pdf
- Code
- DandyQi/CentraBert
- Data
- GLUE, MRPC, MultiNLI, QNLI, SST, SST-2