Federated Continual Learning for Text Classification via Selective Inter-client Transfer
Yatin Chaudhary, Pranav Rai, Matthias Schubert, Hinrich Schütze, Pankaj Gupta
Abstract
In this work, we combine the two paradigms: Federated Learning (FL) and Continual Learning (CL) for text classification task in cloud-edge continuum. The objective of Federated Continual Learning (FCL) is to improve deep learning models over life time at each client by (relevant and efficient) knowledge transfer without sharing data. Here, we address challenges in minimizing inter-client interference while knowledge sharing due to heterogeneous tasks across clients in FCL setup. In doing so, we propose a novel framework, Federated Selective Inter-client Transfer (FedSeIT) which selectively combines model parameters of foreign clients. To further maximize knowledge transfer, we assess domain overlap and select informative tasks from the sequence of historical tasks at each foreign client while preserving privacy. Evaluating against the baselines, we show improved performance, a gain of (average) 12.4% in text classification over a sequence of tasks using five datasets from diverse domains. To the best of our knowledge, this is the first work that applies FCL to NLP.- Anthology ID:
- 2022.findings-emnlp.353
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4789–4799
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.findings-emnlp.353/
- DOI:
- 10.18653/v1/2022.findings-emnlp.353
- Cite (ACL):
- Yatin Chaudhary, Pranav Rai, Matthias Schubert, Hinrich Schütze, and Pankaj Gupta. 2022. Federated Continual Learning for Text Classification via Selective Inter-client Transfer. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4789–4799, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Federated Continual Learning for Text Classification via Selective Inter-client Transfer (Chaudhary et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2022.findings-emnlp.353.pdf