@inproceedings{wang-etal-2025-understanding,
    title = "Understanding the Language Model to Solve the Symbolic Multi-Step Reasoning Problem from the Perspective of Buffer Mechanism",
    author = "Wang, Zhiwei  and
      Wang, Yunji  and
      Zhang, Zhongwang  and
      Zhou, Zhangchen  and
      Jin, Hui  and
      Hu, Tianyang  and
      Sun, Jiacheng  and
      Li, Zhenguo  and
      Zhang, Yaoyu  and
      Xu, Zhi-Qin John",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.893/",
    doi = "10.18653/v1/2025.findings-emnlp.893",
    pages = "16446--16474",
    ISBN = "979-8-89176-335-7",
    abstract = "Large language models have consistently struggled with complex reasoning tasks, such as mathematical problem-solving. Investigating the internal reasoning mechanisms of these models can help us design better model architectures and training strategies, ultimately enhancing their reasoning capability. In this study, we constructed a symbolic multi-step reasoning task to investigate the information propagation mechanisms in Transformer models when solving the task through direct answering and Chain-of-Thought (CoT) reasoning. We introduced the concept of buffer mechanism: the model stores various information in distinct buffers and selectively extracts it through the query-key matrix. We proposed a random matrix-based algorithm to enhance the model{'}s reasoning ability. This algorithm introduces only 132 trainable parameters, yet leads to significant performance improvements on 7 multi-step reasoning datasets, including PrOntoQA, LogicAsker, and LogicInference. These findings provide new insights into understanding the large language models."
}