@inproceedings{wu-etal-2024-conceptmath,
    title = "{C}oncept{M}ath: A Bilingual Concept-wise Benchmark for Measuring Mathematical Reasoning of Large Language Models",
    author = "Wu, Yanan  and
      Liu, Jie  and
      Bu, Xingyuan  and
      Liu, Jiaheng  and
      Zhou, Zhanhui  and
      Zhang, Yuanxing  and
      Zhang, Chenchen  and
      ZhiqiBai, ZhiqiBai  and
      Chen, Haibin  and
      Ge, Tiezheng  and
      Ouyang, Wanli  and
      Su, Wenbo  and
      Zheng, Bo",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2024.findings-acl.407/",
    doi = "10.18653/v1/2024.findings-acl.407",
    pages = "6815--6839",
    abstract = "This paper introduces ConceptMath, a bilingual (English and Chinese), fine-grained benchmark that evaluates concept-wise mathematical reasoning of Large Language Models (LLMs). Unlike traditional benchmarks that evaluate general mathematical reasoning with an average accuracy, ConceptMath systemically organizes math problems under a hierarchy of math concepts, so that mathematical reasoning can be evaluated at different granularity with concept-wise accuracies. Based on our ConcepthMath, we then evaluate a broad range of LLMs, and we observe existing LLMs, though achieving high average accuracies on traditional benchmarks, exhibit significant performance variations across different math concepts and may even fail catastrophically on the most basic ones. Besides, we also introduce an efficient fine-tuning strategy to enhance the weaknesses of existing LLMs. Finally, we hope ConceptMath could guide the developers to understand the fine-grained mathematical abilities of their models and facilitate the growth of foundation models. Code is available at https://github.com/conceptmath/conceptmath."
}