@inproceedings{lundin-etal-2026-guidelines,
    title = "From Guidelines to Guarantees: A Graph-Based Evaluation Harness for Domain-Specific Evaluation of {LLM}s",
    author = "Lundin, Jessica M.  and
      Nakakana, Usman Nasir  and
      Chabot-Couture, Guillaume",
    editor = "Akhtar, Mubashara  and
      Batzner, Jan  and
      Choshen, Leshem  and
      Ghosh, Avijit  and
      Gohar, Usman  and
      Mickel, Jennifer  and
      Pant, Ichhya  and
      Talat, Zeerak  and
      Lin, Michelle",
    booktitle = "Proceedings of the Workshop on Evaluating Evaluations ({E}val{E}val)",
    month = jul,
    year = "2026",
    address = "San Diego, CA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl-workshops/2026.evaleval-1.34/",
    pages = "211--220",
    ISBN = "979-8-89176-429-3",
    abstract = "Rigorous evaluation of domain-specific language models requires benchmarks that are comprehensive, contamination-resistant, and maintainable. Static, manually curated datasets do not satisfy these properties. We present a graph-based evaluation harness that transforms structured clinical guidelines into a queryable knowledge graph and dynamically instantiates evaluation queries via graph traversal. The framework provides three guarantees: (1) complete coverage of guideline relationships; (2) surface-form contamination resistance through combinatorial variation; and (3) validity inherited from expert-authored graph structure. Applied to the WHO IMCI guidelines, the harness generates clinically grounded multiple-choice questions spanning symptom recognition, treatment, severity classification, and follow-up care. Evaluation across five language models reveals systematic capability gaps. Models perform well on symptom recognition but show lower accuracy on treatment protocols and clinical management decisions. The framework supports continuous regeneration of evaluation data as guidelines evolve and generalizes to domains with structured decision logic. This provides a scalable foundation for evaluation infrastructure. Data and Code Availability The WHO IMCI handbook is publicly available (WHO, 2014). Our graph construction, question generation code, and generated question dataset are available at https://github.com/jessicalundin/ graph{\_}testing{\_}harness."
}