@inproceedings{zhang-etal-2025-dorm,
    title = "{DORM}: Preference Data Weights Optimization for Reward Modeling in {LLM} Alignment",
    author = "Zhang, Rongzhi  and
      Zhang, Chenwei  and
      Zhang, Xinyang  and
      Qiu, Liang  and
      Jiang, Haoming  and
      Zhuang, Yuchen  and
      Zhang, Qingru  and
      Yun, Hyokun  and
      Li, Xian  and
      Yin, Bing  and
      Zhao, Tuo  and
      Zhang, Chao",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1237/",
    doi = "10.18653/v1/2025.findings-emnlp.1237",
    pages = "22721--22739",
    ISBN = "979-8-89176-335-7",
    abstract = "Aligning large language models (LLMs) with human preferences relies heavily on high-quality reward models. However, existing approaches struggle with two critical challenges: noisy preference labels and the varying importance of preference samples. We introduce DORM, a method that enhances reward modeling by learning to dynamically weigh preference data.DORM initializes data importance using a combination of model uncertainty and prediction disagreement, then iteratively refines them via bilevel optimization to maximize validation performance. Using only 50k samples, DORM trains a 12B reward model that achieves 90.5{\%} accuracy on RewardBench, matching the performance of models trained on significantly larger datasets. Furthermore, downstream alignment tasks show that fine-tuned LLMs with DORM achieve a 61.2{\%} win rate against baseline methods, highlighting its data efficiency and generalizability."
}