@inproceedings{huang-etal-2025-routereval,
    title = "{R}outer{E}val: A Comprehensive Benchmark for Routing {LLM}s to Explore Model-level Scaling Up in {LLM}s",
    author = "Huang, Zhongzhan  and
      Ling, Guoming  and
      Lin, Yupei  and
      Chen, Yandong  and
      Zhong, Shanshan  and
      Wu, Hefeng  and
      Lin, Liang",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.208/",
    doi = "10.18653/v1/2025.findings-emnlp.208",
    pages = "3860--3887",
    ISBN = "979-8-89176-335-7",
    abstract = "Routing large language models (LLMs) is a new paradigm that uses a router to recommend the best LLM from a pool of candidates for a given input. In this paper, our comprehensive analysis with more than 8,500 LLMs reveals a novel model-level scaling up phenomenon in Routing LLMs, i.e., a capable router can significantly enhance the performance of this paradigm as the number of candidates increases. This improvement can even surpass the performance of the best single model in the pool and many existing strong LLMs, confirming it a highly promising paradigm. However, the lack of comprehensive and open-source benchmarks for Routing LLMs has hindered the development of routers. In this paper, we introduce RouterEval, a benchmark tailored for router research, which includes over 200,000,000 performance records for 12 popular LLM evaluations across various areas such as commonsense reasoning, semantic understanding, etc., based on over 8,500 various LLMs. Using RouterEval, extensive evaluations of existing Routing LLM methods reveal that most still have significant room for improvement."
}