Jieren Deng


A Secure and Efficient Federated Learning Framework for NLP
Chenghong Wang | Jieren Deng | Xianrui Meng | Yijue Wang | Ji Li | Sheng Lin | Shuo Han | Fei Miao | Sanguthevar Rajasekaran | Caiwen Ding
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

In this work, we consider the problem of designing secure and efficient federated learning (FL) frameworks for NLP. Existing solutions under this literature either consider a trusted aggregator or require heavy-weight cryptographic primitives, which makes the performance significantly degraded. Moreover, many existing secure FL designs work only under the restrictive assumption that none of the clients can be dropped out from the training protocol. To tackle these problems, we propose SEFL, a secure and efficient federated learning framework that (1) eliminates the need for the trusted entities; (2) achieves similar and even better model accuracy compared with existing FL designs; (3) is resilient to client dropouts.

TAG: Gradient Attack on Transformer-based Language Models
Jieren Deng | Yijue Wang | Ji Li | Chenghong Wang | Chao Shang | Hang Liu | Sanguthevar Rajasekaran | Caiwen Ding
Findings of the Association for Computational Linguistics: EMNLP 2021

Although distributed learning has increasingly gained attention in terms of effectively utilizing local devices for data privacy enhancement, recent studies show that publicly shared gradients in the training process can reveal the private training data (gradient leakage) to a third-party. We have, however, no systematic understanding of the gradient leakage mechanism on the Transformer based language models. In this paper, as the first attempt, we formulate the gradient attack problem on the Transformer-based language models and propose a gradient attack algorithm, TAG, to reconstruct the local training data. Experimental results on Transformer, TinyBERT4, TinyBERT6 BERT_BASE, and BERT_LARGE using GLUE benchmark show that compared with DLG, TAG works well on more weight distributions in reconstructing training data and achieves 1.5x recover rate and 2.5x ROUGE-2 over prior methods without the need of ground truth label. TAG can obtain up to 90% data by attacking gradients in CoLA dataset. In addition, TAG is stronger than previous approaches on larger models, smaller dictionary size, and smaller input length. We hope the proposed TAG will shed some light on the privacy leakage problem in Transformer-based NLP models.