@inproceedings{wang-etal-2026-mtp,
    title = "{MTP}-{RL}: Acceleration of Reinforcement Learning Rollouts with Policy-Aligned Multi-Token Prediction",
    author = "Wang, Ke  and
      Zeng, Aohan  and
      Du, Zhengxiao  and
      Hu, Yuxuan  and
      Zhang, Bohan  and
      Wang, Xinyi  and
      Tang, Jie  and
      Zhang, Jing",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1871/",
    pages = "37530--37542",
    ISBN = "979-8-89176-395-1",
    abstract = "Reinforcement learning (RL) is widely applied to boost the performance of pretrained models, yet its training efficiency is severely constrained by rollout generation. While speculative decoding based on multi-token prediction (MTP) offers a potential acceleration pathway, its widespread adoption is hindered by the absence of MTP in vanilla pretrained models and the rapid degradation of the MTP acceptance length in RL training. To address these issues, this paper proposes MTP-RL, a two-stage framework that pioneers effective training of MTPs in RL and accelerates the rollout phase for diverse models. It involves a pipeline to equip the multi-layer parameter-sharing MTP for all models and an innovative advantage-aware MTP optimization strategy to facilitate policy-aligned training of MTPs. Experiments demonstrate that our method not only achieves stable growth of acceptance length during RL training, but also accelerates RL rollouts, achieving an average 23.1{\%}{--}55.3{\%} reduction in rollout time compared to baselines."
}