@inproceedings{li-etal-2025-self-improvement,
    title = "Self-Improvement Towards {P}areto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignment",
    author = "Li, Moxin  and
      Zhang, Yuantao  and
      Wang, Wenjie  and
      Shi, Wentao  and
      Liu, Zhuo  and
      Feng, Fuli  and
      Chua, Tat-Seng",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.574/",
    pages = "11010--11031",
    ISBN = "979-8-89176-256-5",
    abstract = "Multi-Objective Alignment (MOA) aims to align LLMs' responses with multiple human preference objectives, with Direct Preference Optimization (DPO) emerging as a prominent approach. However, we find that DPO-based MOA approaches suffer from widespread preference conflicts in the data, where different objectives favor different responses. This results in conflicting optimization directions, hindering the optimization on the Pareto Front. To address this, we propose to construct Pareto-optimal responses to resolve preference conflicts. To efficiently obtain and utilize such responses, we propose a self-improving DPO framework that enables LLMs to self-generate and select Pareto-optimal responses for self-supervised preference alignment. Extensive experiments on two datasets demonstrate the superior Pareto Front achieved by our framework compared to various baselines"
}