@inproceedings{aoki-etal-2024-first,
    title = "First Heuristic Then Rational: Dynamic Use of Heuristics in Language Model Reasoning",
    author = "Aoki, Yoichi  and
      Kudo, Keito  and
      Kuribayashi, Tatsuki  and
      Sone, Shusaku  and
      Taniguchi, Masaya  and
      Sakaguchi, Keisuke  and
      Inui, Kentaro",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.emnlp-main.789/",
    doi = "10.18653/v1/2024.emnlp-main.789",
    pages = "14255--14271",
    abstract = "Explicit multi-step reasoning, such as chain-of-thought, is widely adopted in the community to explore the better performance of language models (LMs). We report on the systematic strategy that LMs use in this process.Our controlled experiments reveal that LMs rely more heavily on heuristics, such as lexical overlap, in the earlier stages of reasoning when more steps are required to reach an answer. Conversely, their reliance on heuristics decreases as LMs progress closer to the final answer. This suggests that LMs track only a limited number of future steps and dynamically combine heuristic strategies with rational ones in solving tasks involving multi-step reasoning."
}