@inproceedings{yu-etal-2025-training,
    title = "Training Language Model to Critique for Better Refinement",
    author = "Yu, Tianshu  and
      Xiang, Chao  and
      Yang, Mingchuan  and
      Ke, Pei  and
      Wen, Bosi  and
      Wang, Cunxiang  and
      Cheng, Jiale  and
      Zhang, Li  and
      Mu, Xinyu  and
      Sun, Chuxiong  and
      Huang, Minlie",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.1373/",
    pages = "26760--26804",
    ISBN = "979-8-89176-256-5",
    abstract = "Large language models (LLMs) have demonstrated remarkable evaluation and critique capabilities, providing insightful feedback and identifying flaws in various tasks. However, limited research has explored which types of critiques are most effective for improving model responses or how to generate such critiques. To address this gap, we introduce Refinement-oriented Critique Optimization (RCO), a novel framework designed to train critic models using refinement signals. RCO uses a feedback loop where critiques, generated by the critic model, guide the actor model in refining its responses. The critique utility (CU) quantifies the effectiveness of these refinements, serving as the reward signal for training the critic model. By focusing on critiques that lead to better refinements, RCO eliminates the need for direct critique preference assessment, ensuring that critiques driving meaningful improvements are rewarded. We evaluate RCO across five tasks{---}dialog generation, summarization, question answering, mathematical reasoning, and code generation{---}and show that it significantly outperforms traditional methods and open-source models in terms of critique quality and refinement outcomes. Our contributions include the introduction of RCO, a novel supervision scheme based on refined response preferences, and comprehensive experimental results that highlight the method{'}s effectiveness in enhancing LLM critique-refinement loops. Code and data will be publicly available upon acceptance of this paper."
}