@inproceedings{coca-etal-2025-aspera,
    title = "{ASPERA}: A Simulated Environment to Evaluate Planning for Complex Action Execution",
    author = "Coca, Alexandru  and
      Gaynor, Mark  and
      Zhang, Zhenxing  and
      Cheng, Jianpeng  and
      Tseng, Bo-Hsiang  and
      Boothroyd, Peter  and
      Martinez Alonso, Hector  and
      O Seaghdha, Diarmuid  and
      Johannsen, Anders",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1234/",
    pages = "25399--25434",
    ISBN = "979-8-89176-251-0",
    abstract = "This work evaluates the potential of large language models (LLMs) to power digital assistants capable of complex action execution. Such assistants rely on pre-trained programming knowledge to execute multi-step goals by composing objects and functions defined in assistant libraries into action execution programs. To achieve this, we develop ASPERA, a framework comprising an assistant library simulation and a human-assisted LLM data generation engine. Our engine allows developers to guide LLM generation of high-quality tasks consisting of complex user queries, simulation state and corresponding validation programs, tackling data availability and evaluation robustness challenges. Alongside the framework we release Asper-Bench, an evaluation dataset of 250 challenging tasks generated using ASPERA, which we use to show that program generation grounded in custom assistant libraries is a significant challenge to LLMs compared to dependency-free code generation."
}