@inproceedings{pan-etal-2025-lemma,
    title = "{LEMMA}: Learning from Errors for {M}athe{M}atical Advancement in {LLM}s",
    author = "Pan, Zhuoshi  and
      Li, Yu  and
      Lin, Honglin  and
      Pei, Qizhi  and
      Tang, Zinan  and
      Wu, Wei  and
      Ming, Chenlin  and
      Zhao, H. Vicky  and
      He, Conghui  and
      Wu, Lijun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.605/",
    pages = "11615--11639",
    ISBN = "979-8-89176-256-5",
    abstract = "Large language models (LLMs) have demonstrated remarkable reasoning capability in solving mathematical problems. However, existing approaches primarily focus on improving the quality of correct training data, e.g., distilling high-quality correct solutions from advanced models, neglecting the value contained in error data, potentially hindering the model{'}s reflective ability. Though some studies attempted to leverage error data, they often involve complex mechanisms, such as Monte Carlo Tree Search (MCTS) to explore error nodes.In this work, we propose to enhance LLM{'}s reasoning ability by Learning from Errors for MatheMatical Advancement (LEMMA). LEMMA constructs data consists of an incorrect solution with an erroneous step and a reflection connection to a correct solution for fine-tuning. Specifically, we systematically analyze the model-generated error types and introduce an {\_}error-type grounded mistake augmentation{\_} method to collect diverse and representative errors. Correct solutions are either from fixing the errors or generating a fresh start. By fine-tuning on the constructed dataset, the model is able to {\_}self-correct errors autonomously{\_} within the generation process {\_}without relying on external critique models{\_}. Experimental results demonstrate that LEMMA achieves significant performance improvements over other strong models with less than $90k$ data."
}