@inproceedings{chen-etal-2025-towards-coarse,
    title = "Towards Coarse-to-Fine Evaluation of Inference Efficiency for Large Language Models",
    author = "Chen, Yushuo  and
      Tang, Tianyi  and
      Xiang, Erge  and
      Li, Linjiang  and
      Zhao, Xin  and
      Wang, Jing  and
      Chai, Yunpeng  and
      Wen, Ji-Rong",
    editor = "Sun, Maosong  and
      Duan, Peiyong  and
      Liu, Zhiyuan  and
      Xu, Ruifeng  and
      Sun, Weiwei",
    booktitle = "Proceedings of the 24th {C}hina National Conference on Computational Linguistics ({CCL} 2025)",
    month = aug,
    year = "2025",
    address = "Jinan, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://preview.aclanthology.org/ingest-ccl/2025.ccl-1.75/",
    pages = "985--1002",
    abstract = "``In real world, large language models (LLMs) can serve as the assistant to help users accomplish their jobs, and also support the development of advanced applications. For the wide application ofLLMs, the inference efficiency is an essential concern, which has been widely studied in existing work, and numerous optimization algorithms and code libraries have been proposed to improve it.Nonetheless, users still find it challenging to compare the effectiveness of all the above method sand understand the underlying mechanisms. In this work, we propose a coarse-to-fine method that encompasses both experimental and analytical components. This method can be applied across various models and inference libraries. Specifically, we examine four usage scenarios within two practical applications. We further provide both theoretical and empirical fine-grained analyses of each module in the Transformer architecture. Our methods can be a general and invaluable method for researchers to evaluate various code libraries and improve inference strategies across different LLMs. We open-source the supporting dataset, code, and evaluation scripts at the link:https://github.com/RUCAIBox/Inference-Efficiency-Evaluation.''"
}