@inproceedings{gupta-etal-2025-carmo,
    title = "{CARMO}: Dynamic Criteria Generation for Context Aware Reward Modelling",
    author = "Gupta, Taneesh  and
      Shandilya, Shivam  and
      Zhang, Xuchao  and
      Madhavan, Rahul  and
      Ghosh, Supriyo  and
      Bansal, Chetan  and
      Yao, Huaxiu  and
      Rajmohan, Saravan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.findings-acl.114/",
    pages = "2202--2261",
    ISBN = "979-8-89176-256-5",
    abstract = "Reward modeling in large language models is known to be susceptible to reward hacking, causing models to latch onto superficial features such as the tendency to generate lists or unnecessarily long responses. In RLHF, and more generally during post-training, flawed reward signals often lead to outputs that optimize for these spurious correlates instead of genuine quality or correctness. We propose **Carmo (Context-Aware Reward Modeling)**, a novel approach that first generates dynamic, context-relevant criteria to ground the reward model prior to producing reward scores. Unlike prior methods that use static rubrics, Carmo leverages powerful LLMs to adaptively create evaluation criteria, e.g., logical consistency, clarity, and depth, tailored to the user query. Our theoretical analysis shows that such criteria generation can mitigate reward hacking. We further demonstrate how Carmo can be distilled into smaller models, thereby lowering the computational cost of alignment. We establish a new state-of-the-art performance on zero shot setting for generative models, with a 2.1{\%} improvement on Reward Bench. Furthermore, alignment performed on the Carmo-curated preference dataset achieves **22.5{\%} and 21.1{\%} LC-WR ({\%}) and WR ({\%}) on Mistral-Base (7B)**. We release our datasets at [huggingface/CARMO](https://huggingface.co/datasets/Multi-preference-Optimization/CARMO-UltraFeedback)."
}