@inproceedings{li-etal-2025-uncertainty,
    title = "Uncertainty-Aware Iterative Preference Optimization for Enhanced {LLM} Reasoning",
    author = "Li, Lei  and
      Liu, Hehuan  and
      Zhou, Yaxin  and
      Gui, ZhaoYang  and
      Weng, Xudong  and
      Yuan, Yi  and
      Wei, Zheng  and
      Li, Zang",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1169/",
    pages = "23996--24012",
    ISBN = "979-8-89176-251-0",
    abstract = "Direct Preference Optimization (DPO) has recently emerged as an efficient and effective method for aligning large language models with human preferences. However, constructing high-quality preference datasets remains challenging, often necessitating expensive manual or powerful LM annotations. Additionally, standard DPO exhibits suboptimal performance in complex reasoning tasks, such as mathematical and code reasoning. In this paper, we introduce an approach to collect preference pairs through iterative sampling and execution feedback, tailored to the current learning state (\textit{e.g.} well-learned, mis-learned, and unlearned) of the policy model. To alleviate the failures of DPO and improve its applicability in reasoning tasks, we propose , an iterative uncertainty-aware preference optimization method that achieves fine-grained preference control by assessing model confidence. We validate our approach across three reasoning tasks, incorporating five established reasoning datasets and one self-curated dataset. Our experimental results demonstrate an overall improvement of 3.6{\%} over the standard DPO method and show the model exhibits promising generalizability."
}