@inproceedings{wang-etal-2025-code,
    title = "To Code or not to Code? Adaptive Tool Integration for Math Language Models via Expectation-Maximization",
    author = "Wang, Haozhe  and
      Li, Long  and
      Qu, Chao  and
      Xu, Weidi  and
      Zhu, Fengming  and
      Chu, Wei  and
      Lin, Fangzhen",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/transition-to-people-yaml/2025.findings-acl.159/",
    doi = "10.18653/v1/2025.findings-acl.159",
    pages = "3060--3075",
    ISBN = "979-8-89176-256-5",
    abstract = "Recent advances in mathematical problem-solving with language models (LMs) integrate chain-of-thought (CoT) reasoning and code execution to harness their complementary strengths. However, existing hybrid frameworks exhibit a critical limitation: they depend on externally dictated instructions or rigid code-integration templates, lacking metacognitive awareness{---}the capacity to dynamically evaluate intrinsic capabilities and autonomously determine when and how to integrate tools. This rigidity motivates our study of autonomous code integration, enabling models to adapt tool-usage strategies as their reasoning abilities evolve during training.While reinforcement learning (RL) shows promise for boosting LLM reasoning at scale (e.g., DeepSeek-R1), we demonstrate its inefficiency in learning autonomous code integration due to inadequate exploration of the vast combinatorial space of CoT-code interleaving patterns. To address this challenge, we propose a novel Expectation-Maximization (EM) framework that synergizes structured exploration (E-step) with off-policy RL optimization (M-step), creating a self-reinforcing cycle between metacognitive tool-use decisions and evolving capabilities. Experiments reveal our method achieves superior results through improved exploration. Notably, our 7B model improves over 11{\%} on MATH500 and 9.4{\%} on AIME without o1-like CoT."
}