@inproceedings{baeumel-etal-2025-disentangling,
    title = "Disentangling Mathematical Reasoning in {LLM}s: A Methodological Investigation of Internal Mechanisms",
    author = "Baeumel, Tanja  and
      van Genabith, Josef  and
      Ostermann, Simon",
    editor = "Valentino, Marco  and
      Ferreira, Deborah  and
      Thayaparan, Mokanarangan  and
      Ranaldi, Leonardo  and
      Freitas, Andre",
    booktitle = "Proceedings of The 3rd Workshop on Mathematical Natural Language Processing (MathNLP 2025)",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ccl/2025.mathnlp-main.16/",
    doi = "10.18653/v1/2025.mathnlp-main.16",
    pages = "203--217",
    ISBN = "979-8-89176-348-7",
    abstract = "Large language models (LLMs) have demonstrated impressive capabilities, yet their internal mechanisms for handling reasoning-intensive tasks remain underexplored. To advance the understanding of model-internal processing mechanisms, we present an investigation of how LLMs perform arithmetic operations by examining internal mechanisms during task execution. Using early decoding, we trace how next token predictions are constructed across layers. Our experiments reveal that while the models recognize arithmetic tasks early, correct result generation occurs only in the final layers. Notably, models proficient in arithmetic exhibit a clear division of labor between attention and MLP modules, where attention propagates input information and MLP modules aggregate it. This division is absent in less proficient models. Furthermore, successful models appear to process more challenging arithmetic tasks functionally, suggesting reasoning capabilities beyond factual recall."
}