@inproceedings{zhu-etal-2026-vrpo,
    title = "{VRPO}: Rethinking Value Modeling for Robust {RL} under Noisy Supervision in {LLM} Post-Training",
    author = "Zhu, Dingwei  and
      Dou, Shihan  and
      Xi, Zhiheng  and
      Jin, Senjie  and
      Zhang, Guoqiang  and
      Zhang, Jiazheng  and
      Ye, Junjie  and
      Chai, Mingxu  and
      Zhou, Enyu  and
      Zhang, Ming  and
      Wang, Yuhui  and
      Huang, Caishuang  and
      Huang, Chenhao  and
      Zhang, Yunke  and
      Wang, Yuran  and
      Gui, Tao  and
      Zhang, Qi  and
      Qiu, Xipeng  and
      Huang, Xuanjing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1103/",
    pages = "24046--24067",
    ISBN = "979-8-89176-390-6",
    abstract = "Reinforcement Learning (RL) in real-world environments often suffers from ambiguous or incomplete reward supervision, which undermines policy stability and generalization. Such noise may cause models to ignore key information or even collapse in advantage estimation. We find that a strong value model is essential for absorbing unstable signals and producing reliable advantages, offering denser and more robust supervision than the reward model. To better optimize noisy supervision, we propose VRPO, a framework that enhances value modeling for robust RL in LLM post-training. VRPO integrates (1) auxiliary losses guided by entropy and perplexity from a frozen language model, and (2) a variational information bottleneck, enabling the value model to filter noise and capture key words. This design allows the value model to correct noise rewards and generate more reliable advantage estimates, transforming it from a passive predictor into an active noise regulator. Experiments on multi-turn dialogue, math reasoning, and science QA with both rule-based and model-based rewards show that VRPO consistently outperforms baselines such as PPO and GRPO. Our work highlight the central role of the value model in Robust RL and provide a principled and practical approach to policy optimization under noisy supervision."
}