@inproceedings{huang-etal-2025-capabilities,
    title = "From Capabilities to Performance: Evaluating Key Functional Properties of {LLM} Architectures in Penetration Testing",
    author = "Huang, Lanxiao  and
      Dave, Daksh  and
      Cody, Tyler  and
      Beling, Peter A.  and
      Jin, Ming",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.802/",
    pages = "15890--15916",
    ISBN = "979-8-89176-332-6",
    abstract = "Large Language Models (LLMs) have been explored for automating or enhancing penetration testing tasks, but their effectiveness and reliability across diverse attack phases remain open questions. This study presents a comprehensive evaluation of multiple LLM-based agents, ranging from singular to modular designs, across realistic penetration testing scenarios, analyzing their empirical performance and recurring failure patterns. We further investigate the impact of core functional capabilities on agent success, operationalized through five targeted augmentations: Global Context Memory (GCM), Inter-Agent Messaging (IAM), Context-Conditioned Invocation (CCI), Adaptive Planning (AP), and Real-Time Monitoring (RTM). These interventions respectively support the capabilities of Context Coherence {\&} Retention, Inter-Component Coordination {\&} State Management, Tool Usage Accuracy {\&} Selective Execution, Multi-Step Strategic Planning {\&} Error Detection {\&} Recovery, and Real-Time Dynamic Responsiveness. Our findings reveal that while some architectures natively exhibit select properties, targeted augmentations significantly enhance modular agent performance{---}particularly in complex, multi-step, and real-time penetration testing scenarios."
}