@inproceedings{li-etal-2025-2d,
    title = "2{D}-{DPO}: Scaling Direct Preference Optimization with 2-Dimensional Supervision",
    author = "Li, Shilong  and
      He, Yancheng  and
      Huang, Hui  and
      Bu, Xingyuan  and
      Liu, Jiaheng  and
      Guo, Hangyu  and
      Wang, Weixun  and
      Gu, Jihao  and
      Su, Wenbo  and
      Zheng, Bo",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.455/",
    pages = "8149--8173",
    ISBN = "979-8-89176-195-7",
    abstract = "Recent advancements in Direct Preference Optimization (DPO) have significantly enhanced the alignment of Large Language Models (LLMs) with human preferences, owing to its simplicity and effectiveness. However, existing methods typically optimize a scalar score or ranking reward, thereby overlooking the multi-dimensional nature of human preferences. In this work, we propose to extend the preference of DPO to two dimensions: segments and aspects. We first introduce a 2D supervision dataset called HelpSteer-2D. For the segment dimension, we divide the response into sentences and assign scores to each segment. For the aspect dimension, we meticulously design several criteria covering the response quality rubrics. With the 2-dimensional signals as feedback, we develop a 2D-DPO framework, decomposing the overall objective into multi-segment and multi-aspect objectives. Extensive experiments on popular benchmarks demonstrate that 2D-DPO performs better than methods that optimize for scalar or 1-dimensional preferences."
}