@inproceedings{ro-etal-2022-scaling,
    title = "Scaling Language Model Size in Cross-Device Federated Learning",
    author = "Ro, Jae  and
      Breiner, Theresa  and
      McConnaughey, Lara  and
      Chen, Mingqing  and
      Suresh, Ananda  and
      Kumar, Shankar  and
      Mathews, Rajiv",
    booktitle = "Proceedings of the First Workshop on Federated Learning for Natural Language Processing (FL4NLP 2022)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.fl4nlp-1.2",
    doi = "10.18653/v1/2022.fl4nlp-1.2",
    pages = "6--20",
    abstract = "Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks. In this work, we leverage various techniques for mitigating these bottlenecks to train larger language models in cross-device federated learning. With systematic applications of partial model training, quantization, efficient transfer learning, and communication-efficient optimizers, we are able to train a 21M parameter Transformer that achieves the same perplexity as that of a similarly sized LSTM with $\sim10\times$ smaller client-to-server communication cost and 11{\%} lower perplexity than smaller LSTMs commonly studied in literature.",
}