@inproceedings{chan-etal-2025-boosting,
    title = "Boosting Policy and Process Reward Models with {M}onte {C}arlo Tree Search in Open-Domain {QA}",
    author = "Chan, Chi-Min  and
      Xu, Chunpu  and
      Zhu, Junqi  and
      Ji, Jiaming  and
      Hong, Donghai  and
      Wen, Pengcheng  and
      Jiang, Chunyang  and
      Ye, Zhen  and
      Yang, Yaodong  and
      Xue, Wei  and
      Han, Sirui  and
      Guo, Yike",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.388/",
    pages = "7433--7451",
    ISBN = "979-8-89176-256-5",
    abstract = "The recent introduction of OpenAI{'}s O1/O3 model represents a significant milestone in developing strong reasoning capabilities in Large Language Models (LLMs). By introducing more computational budget during test-time, LLMs have the potential to explore more accurate and higher-quality solutions. However, such paradigms are primarily verified in domains that have well-defined criteria for responses, such as coding and mathematics. Inspired by the success of this paradigm, we aim to bridge it to more subtle open-domain question answering. Specifically, we utilize search mechanisms such as Monte Carlo Tree Search (MCTS) for both policy model improvement and reward model improvement that achieve better performance in test-time scaling strategies. Our contributions are summarized in two folds: For the training phase, we demonstrate that our approach surpasses previous SOTA automatic data annotation methods and various public instruction-tuning datasets, with fewer data points. This offers a more data-efficient solution for training robust models. For the inference phase, we utilize the intermediate values collected during training data construction to train a process reward model called PRM+. This model employs a novel two-stage training method to provide finer-grained guidance across the generation trajectory. This introduces no additional overhead during training data collection and further enhances performance by scaling test-time computation. Experimental results show that our method can effectively improve the performance of both the policy model and the reward model."
}