@inproceedings{huo-etal-2026-sps,
    title = "{SPS}: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Models",
    author = "Huo, Yifu  and
      Wang, Chenglong  and
      Zhu, Ziming  and
      Xing, Shunjie  and
      Feng, Peinan  and
      Liu, Tongran  and
      He, Qiaozhi  and
      Zhou, Tian Hua  and
      Changxiaojia  and
      Zhu, JingBo  and
      Yu, Zhengtao  and
      Xiao, Tong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.865/",
    pages = "17472--17489",
    ISBN = "979-8-89176-395-1",
    abstract = "Reinforcement learning (RL) has emerged as a promising paradigm for training reasoning-oriented models by leveraging rule-based reward signals. However, RL training typically tends to improve single-sample success rates (i.e., Pass@1) while offering limited exploration of diverse reasoning trajectories, which is crucial for multi-sample performance (i.e., Pass@k). Our preliminary analysis reveals that this limitation stems from a fundamental \textit{squeezing effect}, whereby probability mass is excessively concentrated on a narrow subset of high-reward trajectories, restricting genuine exploration and constraining attainable performance under RL training. To address this issue, in this work, we propose \textbf{S}teering \textbf{P}robability \textbf{S}queezing (SPS), a training paradigm that interleaves conventional RL with inverse reinforcement learning (IRL). SPS treats on-policy rollouts as demonstrations and employs IRL to explicitly reshape the induced trajectory distribution, thereby enhancing exploration without introducing external supervision. Experiments on five commonly used reasoning benchmarks demonstrate that SPS can enable better exploration and improve Pass@k. Beyond algorithmic contributions, we provide an analysis of RL learning dynamics and identify an empirical upper bound on Pass@k, shedding light on intrinsic exploration limits in RL-based reasoning models. Our findings suggest that alternating between RL and IRL offers an effective pathway toward extending the exploration capacity of reasoning-oriented large language models."
}