@inproceedings{sun-etal-2025-disco,
    title = "{D}i{SC}o: Device-Server Collaborative {LLM}-based Text Streaming Services",
    author = "Sun, Ting  and
      Wang, Penghan  and
      Lai, Fan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.734/",
    pages = "14259--14277",
    ISBN = "979-8-89176-256-5",
    abstract = "The rapid rise of large language models (LLMs) in text streaming services has introduced significant cost and Quality of Experience (QoE) challenges in serving millions of daily requests, especially in meeting Time-To-First-Token (TTFT) and Time-Between-Token (TBT) requirements for real-time interactions. Our real-world measurements show that both server-based and on-device deployments struggle to meet diverse QoE demands: server deployments face high costs and last-hop issues (e.g., Internet latency and dynamics), while on-device LLM inference is constrained by resources. We introduce , a device-server cooperative scheduler designed to optimize users' QoE by adaptively routing requests and migrating response generation between endpoints while maintaining cost constraints. employs cost-aware scheduling, leveraging the predictable speed of on-device LLM inference with the flexible capacity of server-based inference to dispatch requests on the fly, while introducing a token-level migration mechanism to ensure consistent token delivery during migration. Evaluations on real-world workloads{---}including commercial services like OpenAI GPT and DeepSeek, and open-source deployments such as LLaMA3{---}show that can improve users' QoE by reducing tail TTFT (11-52{\%}) and mean TTFT (6-78{\%}) across different model-device configurations, while dramatically reducing serving costs by up to 84{\%} through its migration mechanism while maintaining comparable QoE levels."
}