Federated Model Decomposition with Private Vocabulary for Text Classification

Zhuo Zhang, Xiangjing Hu, Lizhen Qu, Qifan Wang, Zenglin Xu

Abstract
With the necessity of privacy protection, it becomes increasingly vital to train deep neural models in a federated learning manner for natural language processing (NLP) tasks. However, recent studies show eavesdroppers (i.e., dishonest servers) can still reconstruct the private input in federated learning (FL). Such a data reconstruction attack relies on the mappings between vocabulary and associated word embedding in NLP tasks, which are unfortunately less studied in current FL methods. In this paper, we propose a fedrated model decomposition method that protects the privacy of vocabularies, shorted as FEDEVOCAB. In FEDEVOCAB, each participant keeps the local embedding layer in the local device and detaches the local embedding parameters from federated aggregation. However, it is challenging to train an accurate NLP model when the private mappings are unknown and vary across participants in a cross-device FL setting. To address this problem, we further propose an adaptive updating technique to improve the performance of local models. Experimental results show that FEDEVOCAB maintains competitive performance and provides better privacy-preserving capacity compared to status quo methods.
Anthology ID:
2022.emnlp-main.430
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
6413–6425
Language:
URL:
https://aclanthology.org/2022.emnlp-main.430
DOI:
10.18653/v1/2022.emnlp-main.430
Bibkey:
Cite (ACL):
Zhuo Zhang, Xiangjing Hu, Lizhen Qu, Qifan Wang, and Zenglin Xu. 2022. Federated Model Decomposition with Private Vocabulary for Text Classification. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6413–6425, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Federated Model Decomposition with Private Vocabulary for Text Classification (Zhang et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2022.emnlp-main.430.pdf
PDF Search