@inproceedings{kate-etal-2026-good,
    title = "How Good Are {LLM}s at Processing Tool Outputs?",
    author = "Kate, Kiran  and
      Rizk, Yara  and
      Ghosh, Poulami  and
      Gulati, Ashu  and
      Chakraborti, Tathagata  and
      Wright, Zidane  and
      Agarwal, Mayank",
    editor = "Demberg, Vera  and
      Inui, Kentaro  and
      Marquez, Llu{\'i}s",
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.134/",
    pages = "2918--2941",
    ISBN = "979-8-89176-380-7",
    abstract = "Most realistic task automation problems require large language models (LLMs) to call tools, which often return complex JSON responses. These responses must be further processed to derive the information necessary for task completion. The ability of LLMs to do so is under-studied. In this paper, we study the tool response processing task and LLMs' abilities to process structured (JSON) responses. We created a dataset for this task, and evaluated 15 open and closed weight models using multiple prompting approaches. Our results show that JSON processing remains a difficult task even for frontier models across multiple prompting strategies. The optimal response processing strategy depends on both the nature and size of the tool outputs, as well as the complexity of the required reasoning. Variations in processing approaches can lead to performance differences ranging from 3{\%} to 50{\%}."
}