@inproceedings{ma-etal-2026-tspo,
    title = "{TSPO}: Breaking the Double Homogenization Dilemma in Multi-turn Search Policy Optimization",
    author = "Ma, Shichao  and
      Ma, Zhiyuan  and
      Yang, Ming  and
      Li, Xiaofan  and
      Wu, Xing  and
      Du, Jintao  and
      Cheng, Yu  and
      Wang, Weiqiang  and
      Liu, Qiliang  and
      Zhou, Zhengyang  and
      Wang, Yang",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1428/",
    pages = "28607--28623",
    ISBN = "979-8-89176-395-1",
    abstract = "Multi-turn tool-integrated reasoning enables Large Language Models (LLMs) to solve complex tasks through iterative information retrieval. However, current reinforcement learning (RL) frameworks for search-augmented reasoning predominantly rely on sparse outcome-level rewards, leading to a ``Double Homogenization Dilemma.'' This manifests as (1) Process homogenization, where the thinking, reasoning, and tooling involved in generation are ignored. (2) Intra-group homogenization, coarse-grained outcome rewards often lead to inefficiencies in intra-group advantage estimation with methods like Group Relative Policy Optimization (GRPO) during sampling. To address this, we propose Turn-level Stage-aware Policy Optimization (TSPO). TSPO introduces the First-Occurrence Latent Reward (FOLR) mechanism, allocating partial rewards to the step where the ground-truth answer first appears, thereby preserving process-level signals and increasing reward variance within groups without requiring external reward models or any annotations. Extensive experiments demonstrate that TSPO significantly outperforms state-of-the-art baselines, achieving average performance gains of 24{\%} and 13.6{\%} on Qwen2.5-3B and 7B models, respectively. Code is available at https://github.com/Flipped-May/TSPO."
}