@inproceedings{chen-etal-2024-autoprm,
    title = "{A}uto{PRM}: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition",
    author = "Chen, Zhaorun  and
      Zhao, Zhuokai  and
      Zhu, Zhihong  and
      Zhang, Ruiqi  and
      Li, Xiang  and
      Raj, Bhiksha  and
      Yao, Huaxiu",
    editor = "Duh, Kevin  and
      Gomez, Helena  and
      Bethard, Steven",
    booktitle = "Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2024.naacl-long.73/",
    doi = "10.18653/v1/2024.naacl-long.73",
    pages = "1346--1362",
    abstract = "Recent advancements in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet their reliance on extensive manual labeling to provide procedural feedback remains a significant impediment. To address this challenge, in this paper, we propose a novel self-supervised framework **AutoPRM** that efficiently enhances the fine-tuning of LLMs for intricate reasoning challenges. Specifically, **AutoPRM** first decomposes complex problems into more manageable subquestions with a controllable granularity switch, then sequentially apply reinforcement learning to iteratively improve the subquestion solver. Additionally, we propose context-guided decoding to avoid reward tampering and guide the subquestion solver towards the solution of the holistic problem. Extensive experiments show that **AutoPRM** significantly improves performance on mathematical and commonsense reasoning tasks over SOTA. More encouragingly, **AutoPRM** can be easily integrated with other orthogonal reasoning pipelines."
}