@inproceedings{yang-etal-2025-evaluating,
    title = "Evaluating Robustness of {LLM}s to Numerical Variations in Mathematical Reasoning",
    author = "Yang, Yuli  and
      Yamada, Hiroaki  and
      Tokunaga, Takenobu",
    editor = "Drozd, Aleksandr  and
      Sedoc, Jo{\~a}o  and
      Tafreshi, Shabnam  and
      Akula, Arjun  and
      Shu, Raphael",
    booktitle = "The Sixth Workshop on Insights from Negative Results in NLP",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.insights-1.16/",
    pages = "171--180",
    ISBN = "979-8-89176-240-4",
    abstract = "Evaluating an LLM`s robustness against numerical perturbation is a good way to know if the LLM actually performs reasoning or just replicates patterns learned. We propose a novel method to augment math word problems (MWPs), producing numerical variations at a large scale utilizing templates. We also propose an automated error classification framework for scalable error analysis, distinguishing calculation errors from reasoning errors. Our experiments using the methods show LLMs are weak against numerical variations, suggesting they are not fully capable of generating valid reasoning steps, often failing in arithmetic operations."
}