@inproceedings{pan-etal-2026-treerpo,
    title = "d-{T}ree{RPO}: Towards More Reliable Policy Optimization for Diffusion Language Models",
    author = "Pan, Leyi  and
      Tao, Shuchang  and
      Zhai, Yunpeng  and
      Fu, Zheyu  and
      Fang, Liancheng  and
      He, Minghua  and
      Zhang, Lingzhe  and
      Liu, Zhaoyang  and
      Ding, Bolin  and
      Liu, Aiwei  and
      Wen, Lijie",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.994/",
    pages = "21801--21822",
    ISBN = "979-8-89176-390-6",
    abstract = "Reinforcement learning (RL) is pivotal for enhancing the reasoning capabilities of diffusion large language models (dLLMs). However, existing dLLM policy optimization methods suffer from two critical reliability bottlenecks: (1) reward sparsity, arising from coarse or unverifiable signals that impede accurate advantage calculation; and (2) their probability estimates do not account for the gap to the unbiased expectation over all decoding orders, which are intractable to compute. To mitigate these issues, we propose \textit{d}-TreeRPO, a reliable RL framework for dLLMs that leverages tree-structured rollouts and bottom-up advantage computation based on verifiable outcome rewards to provide fine-grained and verifiable step-wise reward signals. Furthermore, we provide a theoretical proof demonstrating that increasing prediction confidence effectively minimizes the gap between unbiased expected prediction probabilities and its single-step forward pass estimate. Guided by this analysis, we introduce a time-scheduled self-distillation loss during training that enhances prediction confidence in later training stages, thereby enabling more accurate probability estimation and better performance. Experiments demonstrate that \textit{d}-TreeRPO outperforms existing baselines and achieves significant improvements across multiple reasoning benchmarks. Specifically, it achieves +86.2{\%} on Sudoku, +51.6{\%} on Countdown, +4.5{\%} on GSM8K, and +5.3{\%} on Math500 compared to the base model."
}