@inproceedings{dong-etal-2026-rl,
    title = "{RL}-{PLUS}: Countering Capability Boundary Collapse of {LLM}s in Reinforcement Learning with Hybrid-policy Optimization",
    author = "Dong, Yihong  and
      Jiang, Xue  and
      Tao, Yongding  and
      Liu, Huanyu  and
      Zhang, Kechi  and
      Mou, Lili  and
      Cao, Rongyu  and
      MA, Yingwei  and
      Chen, Jue  and
      Li, Binhua  and
      Jin, Zhi  and
      Huang, Fei  and
      Li, Yongbin  and
      Li, Ge",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1994/",
    pages = "43029--43053",
    ISBN = "979-8-89176-390-6",
    abstract = "Reinforcement Learning with Verifiable Reward (RLVR) has significantly advanced the complex reasoning abilities of Large Language Models (LLMs). However, it struggles to break through the inherent capability boundaries of the base LLM, due to its essentially on-policy strategy coupled with LLM{'}s immense action space and sparse reward. Critically, RLVR can lead to the capability boundary collapse, narrowing the LLM{'}s problem-solving scope. To address this problem, we propose R-PLUS, a novel hybrid-policy optimization approach for LLMs that synergizes internal exploitation with external data to achieve stronger reasoning capabilities and surpass the boundaries of base models. R-PLUS integrates two core components, i.e., Multiple Importance Sampling to address distributional mismatch from external data, and Exploration-Based Advantage Function to guide the model towards high-value, unexplored reasoning paths. We provide both theoretical analysis and extensive experiments to demonstrate the superiority and generalizability of our approach. Compared with existing RLVR methods, R-PLUS achieves 1) state-of-the-art performance on six math reasoning benchmarks; 2) superior performance on six out-of-distribution reasoning tasks; 3) consistent and significant gains across diverse model families, with average relative improvements up to 69.2{\%}. Moreover, the analysis of Pass@k curves indicates that R-PLUS effectively resolves the capability boundary collapse problem."
}