@inproceedings{schlor-etal-2026-towards,
    title = "Towards Knowledge Graph-Grounded Evaluation of Agentic {LLM}s on Cybersecurity Capture-the-Flag Challenges",
    author = {Schl{\"o}r, Daniel  and
      Bohn, Marius  and
      Wolf, Maximilian  and
      Bergner, Kevin  and
      Goldschmied, Christian  and
      Hotho, Andreas},
    editor = "S{\'e}rasset, Gilles  and
      Gkirtzou, Katerina  and
      Cochez, Michael  and
      Kalo, Jan-Christoph",
    booktitle = "Proceedings of the Knowledge Graphs and Large Language Models Workshop ({KG}-{LLM}) @ {LREC}26",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/revision-workflow/2026.kallm-1.15/",
    doi = "10.63317/4ne74wm45iti",
    pages = "144--154",
    abstract = "Evaluating Large Language Model (LLM) agents on complex multi-step cybersecurity tasks requires structured, reproducible evaluation rubrics. We present BraceGreen, a framework that formalizes Capture-the-Flag (CTF) attack paths as knowledge graphs and uses them as gold-standard rubrics for agentic LLM evaluation. Each node in our knowledge graphs represents an attack step annotated with MITRE ATT{\&}CK tactics, goals, commands, expected outputs, and semantic outcomes, while edges encode prerequisites, dependencies, and alternative paths. Our LangGraph-based evaluation workflow employs LLM-as-judge with chain-of-thought reasoning to semantically compare agent predictions against knowledge graph-encoded alternatives. We contribute a benchmark of 7 CTF machines with knowledge graph annotations, three evaluation modes (command prediction, goal inference, anticipated result), and integration with live machine infrastructure via virtual machines and a MCP server. Our approach bridges the gap between unstructured CTF writeups and graph-structured evaluation rubrics."
}