Abstract
Large language models (LLMs) can solve problems step-by-step.While this chain-of-thought (CoT) reasoning boosts LLMs’ performance, it is unclear if LLMs know when to use CoT and whether those CoT are always necessary to answer the question. This paper shows that LLMs tend to generate redundant calculations and reasoning on a manually constructed math QA dataset, GSM8K-Zero.GSM8K-Zero is constructed such that the questions can be answered without any calculations, but LLMs, including Llama-2 models and Claude-2, tend to generate lengthy and unnecessary calculations to answer the questions.We also conduct experiments to explain why LLMs generate redundant calculations and reasonings.- Anthology ID:
- 2024.eacl-short.15
- Volume:
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Yvette Graham, Matthew Purver
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 161–169
- Language:
- URL:
- https://aclanthology.org/2024.eacl-short.15
- DOI:
- Cite (ACL):
- Cheng-Han Chiang and Hung-yi Lee. 2024. Over-Reasoning and Redundant Calculation of Large Language Models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 2: Short Papers), pages 161–169, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Over-Reasoning and Redundant Calculation of Large Language Models (Chiang & Lee, EACL 2024)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2024.eacl-short.15.pdf