Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks

Zixuan Ke, Bing Liu, Wenhan Xiong, Asli Celikyilmaz, Haoran Li

Abstract
Continual learning (CL) has two main objectives: preventing catastrophic forgetting (CF) and encouraging knowledge transfer (KT). The existing literature mainly focused on overcoming CF. Some work has also been done on KT when the tasks are similar. To our knowledge, only one method has been proposed to learn a sequence of mixed tasks. However, these techniques still suffer from CF and/or limited KT. This paper proposes a new CL method to achieve both. It overcomes CF by isolating the knowledge of each task via discovering a sub-network for it. A soft-masking mechanism is also proposed to preserve the previous knowledge and to enable the new task to leverage the past knowledge to achieve KT. Experiments using classification, generation, information extraction, and their mixture (i.e., heterogeneous tasks) show that the proposed method consistently outperforms strong baselines.
Anthology ID:
2023.findings-emnlp.1008
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
15090–15107
Language:
URL:
https://preview.aclanthology.org/icon-24-ingestion/2023.findings-emnlp.1008/
DOI:
10.18653/v1/2023.findings-emnlp.1008
Bibkey:
Cite (ACL):
Zixuan Ke, Bing Liu, Wenhan Xiong, Asli Celikyilmaz, and Haoran Li. 2023. Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15090–15107, Singapore. Association for Computational Linguistics.
Cite (Informal):
Sub-network Discovery and Soft-masking for Continual Learning of Mixed Tasks (Ke et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/icon-24-ingestion/2023.findings-emnlp.1008.pdf
PDF Search Fix data