@inproceedings{zhang-etal-2026-plug,
    title = "Plug-and-Play Data Module for Code {RL}: Adaptive Ambiguity Replay",
    author = "Zhang, Jianqing  and
      Xia, Wei  and
      Hao, Zhezheng  and
      Wang, Hong  and
      Dong, Hande  and
      Lin, Qiang  and
      Liu, Yang  and
      Cao, Jian  and
      Yang, Qiang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.886/",
    pages = "17865--17875",
    ISBN = "979-8-89176-395-1",
    abstract = "Reinforcement learning (RL) is effective for improving code generation but suffers from data scarcity. While experience replay mitigates this, existing approaches rely on static, in-epoch metrics that overlook training dynamics, often introducing low-utility or outdated data. Analyzing RL dynamics via dataset cartography, we observe that ``ambiguous'' samples, which are vital for model generalization, rapidly migrate to ``easy-to-learn'' regions, diminishing their training value. To address this, we propose Adaptive Ambiguity Replay (A2R) for RL, a plug-and-play module that prioritizes cross-epoch ambiguous samples. To neutralize the noise from stale experiences, A2R incorporates an adaptive importance mechanism based on policy divergence to weigh replayed rollouts. Extensive experiments on nine LLMs (3B{--}14B) demonstrate that A2R outperforms state-of-the-art baselines on real-world code editing tasks across both unseen and learned domains. Our results highlight cross-epoch ambiguity as a key factor for effective replay in RL. Code: https://github.com/TsingZ0/verl-A2R"
}