@inproceedings{yu-etal-2025-self,
    title = "Self-Generated Critiques Boost Reward Modeling for Language Models",
    author = "Yu, Yue  and
      Chen, Zhengxing  and
      Zhang, Aston  and
      Tan, Liang  and
      Zhu, Chenguang  and
      Pang, Richard Yuanzhe  and
      Qian, Yundi  and
      Wang, Xuewei  and
      Gururangan, Suchin  and
      Zhang, Chao  and
      Kambadur, Melanie  and
      Mahajan, Dhruv  and
      Hou, Rui",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.573/",
    pages = "11499--11514",
    ISBN = "979-8-89176-189-6",
    abstract = "Reward modeling is crucial for aligning large language models (LLMs) with human preferences, especially in reinforcement learning from human feedback (RLHF). However, current reward models mainly produce scalar scores and struggle to incorporate critiques in a natural language format. We hypothesize that predicting both critiques and the scalar reward would improve reward modeling ability. Motivated by this, we propose Critic-RM, a framework that improves reward models using self-generated critiques without extra supervision. Critic-RM employs a two-stage process: generating and filtering high-quality critiques, followed by joint fine-tuning on reward prediction and critique generation. Experiments across benchmarks show that Critic-RM improves reward modeling accuracy by 3.7{\%}-7.3{\%} compared to standard reward models and LLM judges, demonstrating strong performance and data efficiency. Additional studies further validate the effectiveness of the generated critiques."
}