@inproceedings{yang-etal-2025-probability,
    title = "Probability-Consistent Preference Optimization for Enhanced {LLM} Reasoning",
    author = "Yang, Yunqiao  and
      Ren, Houxing  and
      Lu, Zimu  and
      Wang, Ke  and
      Shi, Weikang  and
      Zhou, Aojun  and
      Pan, Junting  and
      Zhan, Mingjie  and
      Li, Hongsheng",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.333/",
    pages = "6435--6448",
    ISBN = "979-8-89176-256-5",
    abstract = "Recent advances in preference optimization have demonstrated significant potential for improving mathematical reasoning capabilities in large language models (LLMs). While current approaches leverage high-quality pairwise preference data through outcome-based criteria like answer correctness or consistency, they fundamentally neglect the internal logical coherence of responses. To overcome this, we propose Probability-Consistent Preference Optimization (PCPO), a novel framework that establishes dual quantitative metrics for preference selection: (1) surface-level answer correctness and (2) intrinsic token-level probability consistency across responses. Extensive experiments show that our PCPO consistently outperforms existing outcome-only criterion approaches across a diverse range of LLMs and benchmarks. Our code is publicly available at https://github.com/YunqiaoYang/PCPO."
}