@inproceedings{wang-etal-2025-faster,
    title = "Faster and Better {LLM}s via Latency-Aware Test-Time Scaling",
    author = "Wang, Zili  and
      Zhang, Tianyu  and
      Bai, Haoli  and
      Hou, Lu  and
      Yu, Xianzhi  and
      Liu, Wulong  and
      Xiang, Shiming  and
      Zhu, Lei",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.928/",
    doi = "10.18653/v1/2025.findings-emnlp.928",
    pages = "17124--17137",
    ISBN = "979-8-89176-335-7",
    abstract = "Test-Time Scaling (TTS) has proven effective in improving the performance of Large Language Models (LLMs) during inference. However, existing research has overlooked the efficiency of TTS from a latency-sensitive perspective. Through a latency-aware evaluation of representative TTS methods, we demonstrate that a compute-optimal TTS does not always result in the lowest latency in scenarios where latency is critical. To address this gap and achieve latency-optimal TTS, we propose two key approaches by optimizing the concurrency configurations: (1) branch-wise parallelism, which leverages multiple concurrent inference branches, and (2) sequence-wise parallelism, enabled by speculative decoding. By integrating these two approaches and allocating computational resources properly to each, our latency-optimal TTS enables a 32B model to reach 82.3{\%} accuracy on MATH-500 within 1 minute and a smaller 3B model to achieve 72.4{\%} within 10 seconds. Our work emphasizes the importance of latency-aware TTS and demonstrates its ability to deliver both speed and accuracy in latency-sensitive scenarios."
}