@inproceedings{yu-etal-2025-tooling,
    title = "Tooling or Not Tooling? The Impact of Tools on Language Agents for Chemistry Problem Solving",
    author = "Yu, Botao  and
      Baker, Frazier N.  and
      Chen, Ziru  and
      Herb, Garrett  and
      Gou, Boyu  and
      Adu-Ampratwum, Daniel  and
      Ning, Xia  and
      Sun, Huan",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-naacl.424/",
    pages = "7620--7640",
    ISBN = "979-8-89176-195-7",
    abstract = "To enhance large language models (LLMs) for chemistry problem solving, several LLM-based agents augmented with tools have been proposed, such as ChemCrow and Coscientist. However, their evaluations are narrow in scope, leaving a large gap in understanding the benefits of tools across diverse chemistry tasks. To bridge this gap, we develop ChemAgent, an enhanced chemistry agent over ChemCrow, and conduct a comprehensive evaluation of its performance on both specialized chemistry tasks and general chemistry questions. Surprisingly, ChemAgent does not consistently outperform its base LLMs without tools. Our error analysis with a chemistry expert suggests that: For specialized chemistry tasks, such as synthesis prediction, we should augment agents with specialized tools; however, for general chemistry questions like those in exams, agents' ability to reason correctly with chemistry knowledge matters more, and tool augmentation does not always help."
}