@inproceedings{he-etal-2025-beyond,
    title = "Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning",
    author = "He, Qianxi  and
      Ren, Qingyu  and
      Lei, Shanzhe  and
      Wang, Xuhong  and
      Wang, Yingchun",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/name-variant-enfa-fane/2025.emnlp-main.1385/",
    doi = "10.18653/v1/2025.emnlp-main.1385",
    pages = "27215--27231",
    ISBN = "979-8-89176-332-6",
    abstract = "Recent advancements in large language models (LLMs) have shifted the post-training paradigm from traditional instruction tuning and human preference alignment toward reinforcement learning (RL) focused on reasoning capabilities. However, most current methods rely on rule-based evaluations of answer correctness, overlooking the importance of confidence-aware reasoning, especially for small to medium-sized models. These models often receive rewards for speculative answers without generating coherent reasoning chains. To address this limitation, we propose a novel confidence-based reward model tailored for enhancing STEM reasoning capabilities. Unlike conventional approaches, our model penalizes not only incorrect answers but also low-confidence correct responses, thereby promoting more robust and logically consistent reasoning. We validate the effectiveness of our approach through static evaluations, Best-of-N inference tests, and PPO-based RL training. Our method outperforms several state-of-the-art open-source reward models across diverse STEM benchmarks. We release our codes and model in \url{https://github.com/qianxiHe147/C2RM}."
}