@inproceedings{feng-etal-2025-ipr,
    title = "{IPR}: Intelligent Prompt Routing with User-Controlled Quality-Cost Trade-offs",
    author = "Feng, Aosong  and
      Srinivasan, Balasubramaniam  and
      Zhou, Yun  and
      Xu, Zhichao  and
      Zhou, Kang  and
      Guan, Sheng  and
      Chen, Yueyan  and
      Wu, Xian  and
      Kulkarni, Ninad  and
      Zhang, Yi  and
      Shen, Zhengyuan  and
      Bespalov, Dmitriy  and
      Mishra, Soumya Smruti  and
      Teng, Yifei  and
      Wang, Darren Yow-Bang  and
      Ding, Haibo  and
      Cheong, Lin Lee",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-industry.170/",
    pages = "2484--2498",
    ISBN = "979-8-89176-333-3",
    abstract = "Routing incoming queries to the most cost-effective LLM while maintaining response quality poses a fundamental challenge in optimizing performance-cost trade-offs for large-scale commercial systems.We present IPR{---}a quality-constrained Intelligent Prompt Routing framework that dynamically selects optimal models based on predicted response quality and user-specified tolerance levels.IPR introduces three key innovations: (1) a modular architecture with lightweight quality estimators trained on 1.5M prompts annotated with calibrated quality scores, enabling fine-grained quality prediction across model families; (2) a user-controlled routing mechanism with tolerance parameter $\tau \in [0,1]$ that provides explicit control over quality-cost trade-offs; and (3) an extensible design using frozen encoders with model-specific adapters, reducing new model integration from days to hours. To rigorously train and evaluate IPR, we curate an industrial-level IPR dataset, a comprehensive benchmark containing 1.5 million examples with response quality annotations across 11 LLM candidates.Deployed on a major cloud platform, IPR achieves 43.9{\%} cost reduction while maintaining quality parity with the strongest model in the Claude family and processes requests with sub-150ms latency."
}