Abstract
Federated learning is a training paradigm that learns from multiple distributed users without aggregating data on a centralized server, promising the ability to deploy machine-learning to a diverse population of users without first collecting large, labeled datasets. As federated learning involves averaging gradient updates across a decentralized population, there is a growing need for personalization of federated learning systems (i.e. conversational agents must personalize to individual users and the context of an interaction).In this work, we propose a new direction for personalization research within federated learning, leveraging both personal embeddings and shared context embeddings.We also present an approach to predict these “preference” embeddings, enabling personalization without backpropagation. Compared to state-of-the-art personalization baselines, our approach achieves a 50% improvement in test-time perplexity using 0.001% of the memory required by baseline approaches, and achieving greater sample- and compute-efficiency.- Anthology ID:
- 2023.findings-eacl.64
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 869–882
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.64
- DOI:
- 10.18653/v1/2023.findings-eacl.64
- Cite (ACL):
- Andrew Silva, Pradyumna Tambwekar, and Matthew Gombolay. 2023. FedPerC: Federated Learning for Language Generation with Personal and Context Preference Embeddings. In Findings of the Association for Computational Linguistics: EACL 2023, pages 869–882, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- FedPerC: Federated Learning for Language Generation with Personal and Context Preference Embeddings (Silva et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-eacl.64.pdf