@inproceedings{zang-etal-2025-internlm,
    title = "{I}ntern{LM}-{XC}omposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model",
    author = "Zang, Yuhang  and
      Dong, Xiaoyi  and
      Zhang, Pan  and
      Cao, Yuhang  and
      Liu, Ziyu  and
      Ding, Shengyuan  and
      Wu, Shenxi  and
      Ma, Yubo  and
      Duan, Haodong  and
      Zhang, Wenwei  and
      Chen, Kai  and
      Lin, Dahua  and
      Wang, Jiaqi",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.340/",
    doi = "10.18653/v1/2025.findings-acl.340",
    pages = "6547--6563",
    ISBN = "979-8-89176-256-5",
    abstract = "Despite the promising performance of Large Vision Language Models (LVLMs) in visual understanding, they occasionally generate incorrect outputs. While reward models (RMs) with reinforcement learning or test-time scaling offer the potential for improving generation quality, a critical gap remains: publicly available multi-modal RMs for LVLMs are scarce, and the implementation details of proprietary models are often unclear. We bridge this gap with InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective multi-modal reward model that aligns LVLMs with human preferences. To ensure the robustness and versatility of IXC-2.5-Reward, we set up a high-quality multi-modal preference corpus spanning text, image, and video inputs across diverse domains, such as instruction following, general understanding, text-rich documents, mathematical reasoning, and video understanding. IXC-2.5-Reward achieves excellent results on the latest multi-modal reward model benchmark and shows competitive performance on text-only reward model benchmarks. We further demonstrate three key applications of IXC-2.5-Reward: (1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows consistent improvements in instruction following and multi-modal open-ended dialogue; (2) Selecting the best response from candidate responses for test-time scaling; and (3) Filtering outlier or noisy samples from existing image and video instruction tuning training data."
}