LinguaLinked: Distributed Large Language Model Inference on Mobile Devices
Junchen Zhao, Yurun Song, Simenl3@uci.edu Simenl3@uci.edu, Ian Harris, Sangeetha Abdu Jyothi
Abstract
Deploying Large Language Models (LLMs) locally on mobile devices presents a significant challenge due to their extensive memory requirements. In this paper, we introduce LinguaLinked, a system for decentralized, distributed LLM inference on mobile devices. LinguaLinked enables collaborative execution of the inference task across multiple trusted devices and ensures data privacy by processing information locally. LinguaLinked uses three key strategies. First, an optimized model assignment technique segments LLMs and uses linear optimization to align segments with each device�s capabilities. Second, an optimized data transmission mechanism ensures efficient and structured data flow between model segments while also maintaining the integrity of the original model structure. Finally, LinguaLinked incorporates a runtime load balancer that actively monitors and redistributes tasks among mobile devices to prevent bottlenecks, enhancing the system�s overall efficiency and responsiveness. We demonstrate that LinguaLinked facilitates efficient LLM inference while maintaining consistent throughput and minimal latency through extensive testing across various mobile devices, from high-end to low-end Android devices.- Anthology ID:
- 2024.acl-demos.16
- Volume:
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Yixin Cao, Yang Feng, Deyi Xiong
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 160–171
- Language:
- URL:
- https://aclanthology.org/2024.acl-demos.16
- DOI:
- Cite (ACL):
- Junchen Zhao, Yurun Song, Simenl3@uci.edu Simenl3@uci.edu, Ian Harris, and Sangeetha Abdu Jyothi. 2024. LinguaLinked: Distributed Large Language Model Inference on Mobile Devices. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 160–171, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- LinguaLinked: Distributed Large Language Model Inference on Mobile Devices (Zhao et al., ACL 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.acl-demos.16.pdf