@inproceedings{liu-etal-2025-llms-capable,
    title = "Are Your {LLM}s Capable of Stable Reasoning?",
    author = "Liu, Junnan  and
      Liu, Hongwei  and
      Xiao, Linchen  and
      Wang, Ziyi  and
      Liu, Kuikun  and
      Gao, Songyang  and
      Zhang, Wenwei  and
      Zhang, Songyang  and
      Chen, Kai",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.905/",
    doi = "10.18653/v1/2025.findings-acl.905",
    pages = "17594--17632",
    ISBN = "979-8-89176-256-5",
    abstract = "The rapid advancement of large language models (LLMs) has shown remarkable progress in complex reasoning tasks. However, a significant disparity exists between benchmark performances and real-world applications. We attribute this gap primarily to current evaluation protocols and metrics, which inadequately capture the full spectrum of LLM capabilities, especially in complex reasoning tasks where both accuracy and consistency are essential. In this paper, we introduce **G-Pass@**$k$, a novel evaluation metric that continuously assesses model performance across multiple sampling attempts, quantifying both the model{'}s performance potential and its stability. Through extensive experiments on various public and newly constructed benchmarks, we employ G-Pass@$k$ in conjunction with state-of-the-art large language models to provide comprehensive insights into their potential capabilities and operational consistency. Our findings reveal a significant opportunity to enhance the realistic reasoning abilities of LLMs, underscoring the necessity for more robust evaluation metrics."
}