@inproceedings{melas-kyriazi-wang-2022-intrinsic,
    title = "Intrinsic Gradient Compression for Scalable and Efficient Federated Learning",
    author = "Melas-Kyriazi, Luke  and
      Wang, Franklyn",
    editor = "Lin, Bill Yuchen  and
      He, Chaoyang  and
      Xie, Chulin  and
      Mireshghallah, Fatemehsadat  and
      Mehrabi, Ninareh  and
      Li, Tian  and
      Soltanolkotabi, Mahdi  and
      Ren, Xiang",
    booktitle = "Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2022.fl4nlp-1.4/",
    doi = "10.18653/v1/2022.fl4nlp-1.4",
    pages = "27--41",
    abstract = "Federated learning is a rapidly growing area of research, holding the promise of privacy-preserving distributed training on edge devices. The largest barrier to wider adoption of federated learning is the communication cost of model updates, which is accentuated by the fact that many edge devices are bandwidth-constrained. At the same time, within the machine learning theory community, a separate line of research has emerged around optimizing networks within a subspace of the full space of all parameters. The dimension of the smallest subspace for which these methods still yield strong results is called the intrinsic dimension. In this work, we prove a general correspondence between the notions of intrinsic dimension and gradient compressibility, and we show that a family of low-bandwidth federated learning algorithms, which we call intrinsic gradient compression algorithms, naturally emerges from this correspondence. Finally, we conduct large-scale NLP experiments using transformer models with over 100M parameters (GPT-2 and BERT), and show that our method significantly outperforms the state-of-the-art in gradient compression."
}