@inproceedings{chen-etal-2025-finereason,
    title = "{F}ine{R}eason: Evaluating and Improving {LLM}s' Deliberate Reasoning through Reflective Puzzle Solving",
    author = "Chen, Guizhen  and
      Xu, Weiwen  and
      Zhang, Hao  and
      Chan, Hou Pong  and
      Liu, Chaoqun  and
      Bing, Lidong  and
      Zhao, Deli  and
      Luu, Anh Tuan  and
      Rong, Yu",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.333/",
    pages = "6685--6715",
    ISBN = "979-8-89176-251-0",
    abstract = "Many challenging reasoning tasks require not just rapid, intuitive responses, but a more deliberate, multi-step approach. Recent progress in large language models (LLMs) highlights an important shift from the ``System 1'' way of quick reactions to the ``System 2'' style of reflection-and-correction problem solving. However, current benchmarks heavily rely on the final-answer accuracy, leaving much of a model{'}s intermediate reasoning steps unexamined. This fails to assess the model{'}s ability to reflect and rectify mistakes within the reasoning process. To bridge this gap, we introduce FINEREASON, a logic-puzzle benchmark for systematic evaluation of LLMs' reasoning capabilities. Each puzzle can be decomposed into atomic steps, making it ideal for rigorous validation of intermediate correctness. Building on this, we introduce two tasks: state checking and state transition, for a comprehensive evaluation of how models assess the current situation and plan the next move. To support broader research, we also provide a puzzle training set aimed at enhancing general reasoning. We show that models trained on our state checking and transition data demonstrate gains in mathematical reasoning by up to 5.1{\%}."
}