@inproceedings{dou-etal-2025-lost,
    title = "Lost in the Context: Insufficient and Distracted Attention to Contexts in Preference Modeling",
    author = "Dou, Shihan  and
      Chen, Jiayi  and
      Huang, Chenhao  and
      Chen, Feng  and
      Chengzhi, Wei  and
      Zheng, Huiyuan  and
      Liu, Shichun  and
      Liu, Yan  and
      Liu, Chenxiao  and
      Xin, Chao  and
      Yan, Lin  and
      Zhang, Zongzhang  and
      Gui, Tao  and
      Zhang, Qi  and
      Huang, Xuanjing",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.285/",
    pages = "5710--5728",
    ISBN = "979-8-89176-251-0",
    abstract = "In Reinforcement Learning from Human Feedback (RLHF), the reward model (RM) evaluates the response quality based on the given context and assigns a reward. It plays a crucial role in aligning RLHF with human preferences. Although the current RM training paradigm concatenates the context and response while amplifying the reward difference between good and bad response pairs, we demonstrate that the RM faces two significant issues: i) it often allocates only a small proportion of attention to the context, and ii) it frequently ignores segments of the context that are relevant for evaluating the response quality. These issues undermine the RM{'}s effectiveness in modeling human preferences. To further address these challenges, we propose AttnRM, a novel optimization framework that enables the RM to concentrate on crucial segments of the context. Experimental results demonstrate that AttnRM significantly improves preference modeling by increasing attention to relevant information within the context. It also enhances the RM{'}s generalizability and achieves better performance in aligning with human preferences."
}