@inproceedings{chen-etal-2025-better-process,
    title = "Better Process Supervision with Bi-directional Rewarding Signals",
    author = "Chen, Wenxiang  and
      He, Wei  and
      Xi, Zhiheng  and
      Guo, Honglin  and
      Hong, Boyang  and
      Zhang, Jiazheng  and
      Li, Nijun  and
      Gui, Tao  and
      Li, Yun  and
      Zhang, Qi  and
      Huang, Xuanjing",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.747/",
    pages = "14471--14485",
    ISBN = "979-8-89176-256-5",
    abstract = "Process supervision, i.e., evaluating each step, is critical for complex large language model (LLM) reasoning and test-time searching with increased inference compute. Existing approaches, represented by process reward models (PRMs), primarily focus on rewarding signals up to the current step, exhibiting a one-directional nature and lacking a mechanism to model the distance to the final target. To address this problem, we draw inspiration from the A* algorithm, which states that an effective supervisory signal should simultaneously consider the incurred cost and the estimated cost for reaching the target. Building on this key insight, we introduce BiRM, a novel process supervision model that not only evaluates the correctness of previous steps but also models the probability of future success. We conduct extensive experiments on mathematical reasoning tasks and demonstrate that BiRM provides more precise evaluations of LLM reasoning steps, achieving an improvement of 3.1{\%} on Gaokao2023 over PRM under the Best-of-N sampling method. Besides, in search-based strategies, BiRM provides more comprehensive guidance and outperforms ORM by 5.0{\%} and PRM by 3.8{\%} respectively on MATH-500."
}