@inproceedings{zhang-etal-2025-llama,
    title = "{LL}a{MA}-Berry: Pairwise Optimization for Olympiad-level Mathematical Reasoning via O1-like {M}onte {C}arlo Tree Search",
    author = "Zhang, Di  and
      Wu, Jianbo  and
      Lei, Jingdi  and
      Che, Tong  and
      Li, Jiatong  and
      Xie, Tong  and
      Huang, Xiaoshui  and
      Zhang, Shufei  and
      Pavone, Marco  and
      Li, Yuqiang  and
      Ouyang, Wanli  and
      Zhou, Dongzhan",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.375/",
    pages = "7315--7337",
    ISBN = "979-8-89176-189-6",
    abstract = "This paper presents LLaMA-Berry, an advanced mathematical reasoning framework to enhance the problem-solving ability of large language models (LLMs). The framework combines Monte Carlo Tree Search with Self-Refine (SR-MCTS) to optimize the reasoning paths and utilizes a pairwise reward model to evaluate different paths globally. By leveraging the self-critique and rewriting capabilities of LLMs, our SR-MCTS overcomes the inefficiencies and limitations of conventional step-wise and greedy search algorithms, enabling a more efficient exploration of solution spaces. To guide the search process, we propose the Pairwise Preference Reward Model (PPRM), which predicts pairwise preferences between solutions through instruction-following capabilities trained by Reinforcement Learning from Human Feedback (RLHF). Finally, the Enhanced Borda Count (EBC) method is adopted to synthesize pairwise preferences into global quantile scores for evaluations. This approach mitigates the challenges of scoring variability and non-independent distributions in mathematical reasoning tasks. The framework has been tested on general and advanced benchmarks, showing superior search efficiency and performance compared to existing open-source and closed-source methods, particularly in complex Olympiad-level benchmarks, including AIME24 and AMC23."
}