@inproceedings{hassan-etal-2026-large,
    title = "Large Language Models for {IT} Automation Tasks: Are We There Yet?",
    author = "Hassan, Md. Mahadi  and
      Salvador, John  and
      Rahman, Akond Ashfaque Ur  and
      Karmaker, Santu",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.560/",
    pages = "11534--11573",
    ISBN = "979-8-89176-395-1",
    abstract = "LLMs show promise in code generation, yet their effectiveness for IT automation tasks, particularly for tools like Ansible, remains understudied. Existing benchmarks rely primarily on synthetic tasks that fail to capture the needs of practitioners who use IT automation tools. We present ExITBench (Execution-based IT Automation Benchmark), a benchmark of 126 diverse tasks (e.g., configuring servers and managing files) in which each task captures state reconciliation - a core property of IT automation tools. ExITBench evaluates LLMs' ability to generate functional Ansible automation scripts via dynamic execution in controlled environments. We evaluate 14 open-source and 3 proprietary LLMs and find that GPT-4.1-Mini achieves the best pass@10 rate of 23.9{\%}, while Claude-3.5-Sonnet achieves the best pass@1 performance. To explain the low performance, we analyze 1,517 execution failures across the evaluated LLMs and identify two prevalent semantic error categories: failures in state-reconciliation reasoning (42.117{\%} combined from variable (12.287{\%}), host (10.363{\%}), path (10.511{\%}), and template (8.956{\%}) issues) and deficiencies in module-specific execution knowledge (26.203{\%} combined from attribute {\&} parameter (17.617{\%}) and module (8.586{\%}) errors). Our findings reveal key limitations in LLMs' ability to address state reconciliation and apply specialized module knowledge, indicating that reliable IT automation with LLM-based agents need major advances in state reasoning and domain-specific execution."
}