@inproceedings{liu-etal-2026-quartz,
    title = "{QUARTZ}: Quantile-Aware Routing and Queueing for {TTFT} {SLO}s in {LLM} Serving",
    author = "Liu, Zhipeng  and
      Zheng, Yifan  and
      Kong, Fanqi  and
      Zhao, Ziming",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1888/",
    pages = "37886--37896",
    ISBN = "979-8-89176-395-1",
    abstract = "Large Language Model (LLM) serving systems increasingly face strict time-to-first-token (TTFT) service-level objectives (SLOs), yet TTFT remains highly sensitive to router-side queueing effects. Prefill costs scale with prompt length, decode lengths are uncertain, and prefix locality creates strong performance skew across requests. Despite major advances in continuous batching and KV-cache management, today{'}s routers are often agnostic to request cost, which makes them vulnerable to head-of-line blocking and tail-latency amplification under mixed workloads. We propose \textit{QUARTZ}, a quantile-aware routing and queueing layer for LLM serving that predicts conservative quantile-based request-cost proxies, rather than point estimates, using lightweight router-visible signals. QUARTZ uses these quantiles together with backlog-aware router signals to guide worker selection and admission decisions that better align with TTFT tail SLOs while preserving fairness. We implement QUARTZ as a router upgrade for SGLang and evaluate it on representative interactive and retrieval-augmented workloads. The results show reductions in TTFT tail latency and SLO violations across heterogeneous workloads."
}