@inproceedings{zhang-etal-2025-turnbench,
    title = "{T}urn{B}ench-{MS}: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models",
    author = "Zhang, Yiran  and
      Wang, Mo  and
      Li, Xiaoyang  and
      Ren, Kaixuan  and
      Zhu, Chencheng  and
      Naseem, Usman",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1084/",
    doi = "10.18653/v1/2025.findings-emnlp.1084",
    pages = "19892--19924",
    ISBN = "979-8-89176-335-7",
    abstract = "Despite impressive advances in large language models (LLMs), existing benchmarks often focus on single-turn or single-step tasks, failing to capture the kind of iterative reasoning required in real-world settings. To address this limitation, we introduce **TurnBench**, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task inspired by the ``Turing Machine Board Game.'' In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds. This dynamic setup requires models to reason over time, adapt based on past information, and maintain consistency across steps{---}capabilities underexplored in current benchmarks. TurnBench includes two modes: *Classic*, which tests standard reasoning, and *Nightmare*, which introduces increased complexity and requires robust inferential chains. To support fine-grained analysis, we provide ground-truth annotations for intermediate reasoning steps. Our evaluation of state-of-the-art LLMs reveals significant gaps: the best model achieves 84{\%} accuracy in Classic mode, but performance drops to 18{\%} in Nightmare mode. In contrast, human participants achieve 100{\%} in both, underscoring the challenge TurnBench poses to current models. By incorporating feedback loops and hiding task rules, TurnBench reduces contamination risks and provides a rigorous testbed for diagnosing and advancing multi-step, multi-turn reasoning in LLMs."
}