@inproceedings{milchevski-etal-2025-multi,
    title = "Multi-Step Generation of Test Specifications using Large Language Models for System-Level Requirements",
    author = {Milchevski, Dragan  and
      Frank, Gordon  and
      H{\"a}tty, Anna  and
      Wang, Bingqing  and
      Zhou, Xiaowei  and
      Feng, Zhe},
    editor = "Rehm, Georg  and
      Li, Yunyao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.acl-industry.11/",
    doi = "10.18653/v1/2025.acl-industry.11",
    pages = "132--146",
    ISBN = "979-8-89176-288-6",
    abstract = "System-level testing is a critical phase in the development of large, safety-dependent systems, such as those in the automotive industry. However, creating test specifications can be a time-consuming and error-prone process. This paper presents an AI-based assistant to aid users in creating test specifications for system-level requirements. The system mimics the working process of a test developer by utilizing a LLM and an agentic framework, and by introducing intermediate test artifacts - structured intermediate representations derived from input requirements. Our user study demonstrates a 30 to 40{\%} reduction in effort required for test development. For test specification generation, our quantitative analysis reveals that iteratively providing the model with more targeted information, like examples of similar test specifications, based on comparable requirements and purposes, can boost the performance by up to 30{\%} in ROUGE-L. Overall, our approach has the potential to improve the efficiency, accuracy, and reliability of system-level testing and can be applied to various industries where safety and functionality are paramount."
}