@inproceedings{gioacchini-etal-2025-autopenbench,
    title = "{A}uto{P}en{B}ench: A Vulnerability Testing Benchmark for Generative Agents",
    author = "Gioacchini, Luca  and
      Delsanto, Alexander  and
      Drago, Idilio  and
      Mellia, Marco  and
      Siracusano, Giuseppe  and
      Bifulco, Roberto",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-industry.114/",
    pages = "1615--1624",
    ISBN = "979-8-89176-333-3",
    abstract = "LLM agents show promise for vulnerability testing. We however lack benchmarks to evaluate and compare solutions. AutoPenBench covers this need offering an open benchmark for the evaluation of vulnerability testing agents. It includes 33 tasks, ranging from introductory exercises to actual vulnerable systems. It supports MCP, enabling the comparison of agent capabilities. We introduce milestones per task, allowing the comparison of intermediate steps where agents struggle. To illustrate the use of AutoPenBench we evaluate autonomous and human-assisted agent architectures. The former achieves 21{\%} success rates, insufficient for production, while human-assisted agents reach 64{\%} success, indicating a viable industrial path. AutoPenBench is offered as open source and enables fair comparison of agents."
}