@inproceedings{du-etal-2026-escaping,
    title = "Escaping the Echo Trap: On Credit Assignment Failure in Multi-turn {LLM} Self-Reflection",
    author = "Du, Linxuan  and
      Xue, Guangquan  and
      Liang, Xiaobo  and
      Huang, Qipeng  and
      Ding, Yuyang  and
      Shi, Xinyu  and
      Yijun, Zhang  and
      Qi, Ji  and
      Zhu, Wenpeng  and
      Li, Juntao  and
      Zhang, Min",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.1636/",
    pages = "35393--35405",
    ISBN = "979-8-89176-390-6",
    abstract = "Despite the potential of multi-turn self-reflection to improve LLM reasoning, its effectiveness in practice is severely constrained by a failure mode we term the Echo Trap.Specifically, this phenomenon gives rise to two coupled problems: (1) the model becomes limited by its inherent capabilities and tends to repeat earlier reflections to preserve reward signals; (2) once such ``copy'' behavior is reinforced, the model ceases to try new strategies, leading to exploration collapse.We attribute this issue to imprecise credit assignment during training, as standard GRPO assigns rewards at the trajectory level, making it difficult to distinguish which reflection steps contribute to improved outcomes.To address this limitation, we propose a tree-structured extension of GRPO for multi-turn self-reflection, which enables more accurate advantage estimation.Through extensive experiments, we analyze the Echo Trap and demonstrate that our method effectively mitigates behavior collapse and improves performance across multiple benchmarks."
}