@inproceedings{seegmiller-etal-2025-flames,
    title = "{FLAMES}: Improving {LLM} Math Reasoning via a Fine-Grained Analysis of the Data Synthesis Pipeline",
    author = "Seegmiller, Parker  and
      Mehta, Kartik  and
      Saha, Soumya  and
      Tao, Chenyang  and
      Oraby, Shereen  and
      Gupta, Arpit  and
      Chung, Tagyoung  and
      Bansal, Mohit  and
      Peng, Nanyun",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1346/",
    doi = "10.18653/v1/2025.findings-emnlp.1346",
    pages = "24746--24766",
    ISBN = "979-8-89176-335-7",
    abstract = "Recent works improving LLM math reasoning with synthetic data have used unique setups, making comparison of data synthesis strategies impractical. This leaves many unanswered questions about the roles of different factors in the synthetic data pipeline, such as the impact of filtering low-quality problems. To address this gap, we introduce FLAMES, a Framework for LLM Assessment of Math rEasoning Data Synthesis, and perform a systematic study of 10 existing data synthesis strategies and multiple other factors impacting the performance of synthetic math reasoning data. Our FLAMES experiments provide several valuable insights about the optimal balance of difficulty and diversity of synthetic data. First, data agents designed to increase problem complexity lead to best improvements on most math metrics. Second, with a fixed data generation budget, keeping higher problem coverage is more important than keeping only problems with reliable solutions. Third, GSM8K- and MATH-based synthetic data can lead to improvements on competition-level benchmarks, showcasing easy-to-hard generalization. Leveraging insights from our FLAMES experiments, we design two novel data synthesis strategies for improving out-of-domain generalization and robustness. Further, we develop the FLAMES dataset, an effective blend of our novel and existing data synthesis strategies, outperforming public datasets on OlympiadBench (+15.7), CollegeMath (+4.5), GSMPlus (+6.5), and MATH (+3.1). Fine-tuning Qwen2.5-Math-7B on the FLAMES dataset achieves 81.4{\%} on MATH, surpassing larger Llama3 405B, GPT-4o and Claude 3.5 Sonnet."
}