@inproceedings{cai-etal-2025-k,
    title = "K-order Ranking Preference Optimization for Large Language Models",
    author = "Cai, Shihao  and
      Gao, Chongming  and
      Zhang, Yang  and
      Shi, Wentao  and
      Zhang, Jizhi  and
      Bao, Keqin  and
      Wang, Qifan  and
      Feng, Fuli",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.findings-acl.250/",
    pages = "4844--4859",
    ISBN = "979-8-89176-256-5",
    abstract = "To adapt large language models (LLMs) to ranking tasks, existing list-wise methods, represented by list-wise Direct Preference Optimization (DPO), focus on optimizing partial-order or full-order list ranking consistency for LLMs to enhance their ranking abilities.However, we argue that optimizing top-K ranking consistency could be more appropriate for real-world applications. There are two main reasons: (1) users are typically concerned with only the top-K results, making top-K ranking more important, and (2) tail items often lack precise feedback, making top-K ranking more reliable. Based on this, we propose $\textbf{K}$-order Ranking $\textbf{P}$reference $\textbf{O}$ptimization (KPO) by extending the DPO{'}s Plackett-Luce model to accommodate top-K rankings. Additionally, recognizing that the number of important items can vary across queries, we extend KPO to dynamically determine appropriate $K$ for different samples and introduce a curriculum learning strategy to boost training efficiency. Extensive experiments demonstrate the effectiveness of KPO, highlighting its high sample efficiency and robustness to noise. The code is available at https://github.com/Lanyu0303/KPO."
}