@inproceedings{lyu-etal-2025-enhancing,
    title = "Enhancing Open-Domain Task-Solving Capability of {LLM}s via Autonomous Tool Integration from {G}it{H}ub",
    author = "Lyu, Bohan  and
      Cong, Xin  and
      Yu, Heyang  and
      Yang, Pan  and
      Qian, Cheng  and
      Wang, Zihe  and
      Qin, Yujia  and
      Ye, Yining  and
      Lu, Yaxi  and
      Qian, Chen  and
      Zhang, Zhong  and
      Yan, Yukun  and
      Lin, Yankai  and
      Liu, Zhiyuan  and
      Sun, Maosong",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.845/",
    pages = "17257--17277",
    ISBN = "979-8-89176-251-0",
    abstract = "Large Language Models (LLMs) excel in traditional natural language processing tasks but struggle with problems that require complex domain-specific calculations or simulations. While equipping LLMs with external tools to build LLM-based agents can enhance their capabilities, existing approaches lack the flexibility to address diverse and ever-evolving user queries in open domains. Currently, there is also no existing dataset that evaluates LLMs on open-domain knowledge that requires tools to solve. To this end, we introduce OpenAct benchmark to evaluate the open-domain task-solving capability, which is built on human expert consultation and repositories in GitHub. It comprises 339 questions spanning 7 diverse domains that need to be solved with domain-specific methods. In our experiments, even state-of-the-art LLMs and LLM-based agents demonstrate unsatisfactory success rates, underscoring the need for a novel approach.Furthermore, we present OpenAgent, a novel LLM-based agent system that can tackle evolving queries in open domains through autonomously integrating specialized tools from GitHub. OpenAgent employs 1) a hierarchical framework where specialized agents handle specific tasks and can assign tasks to inferior agents, 2) a bi-level experience learning mechanism to learn from both humans' and its own experiences to tackle tool flaws. Experiments demonstrate its superior effectiveness and efficiency, which significantly outperforms baselines. Our data and code are open-source at https://github.com/OpenBMB/OpenAct."
}