@inproceedings{yang-etal-2026-datasets,
    title = "Datasets for a Chatbot for Clinical Trial Search",
    author = "Yang, Yumeng  and
      Ludmir, Ethan  and
      Roberts, Kirk",
    editor = "Gupta, Deepak  and
      Thompson, Paul  and
      Ananiadou, Sophia  and
      Demner-Fushman, Dina",
    booktitle = "Proceedings of the Third Workshop on Patient-Oriented Language Processing ({CL}4{H}ealth) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/paragraph-normalization/2026.cl4health-1.13/",
    doi = "10.63317/3pk9cracmxy7",
    pages = "139--148",
    abstract = "Matching patients to clinical trials is a critical bottleneck hindered by complex eligibility criteria. While conversational AI offers a promising solution, its safe deployment depends on high-quality, domain specific data. This paper introduces three benchmark datasets designed to support the development and evaluation of conversational agents for clinical trial pre-screening. First, a manually-annotated paired-criterion dataset provides a gold standard for structuring raw criteria, which we used to objectively group 12,596 criteria. Second, we curated a human-authored question benchmark to validate the clinical fidelity and patient-centric clarity of questions generated by a medical LLM, ensuring the AI{'}s dialogue is accurate and understandable. Third, we constructed a human-validated assessment corpus of criterion-question-answer tuples with human-labeled outcomes to evaluate criterion classification based on a patient{'}s answer to a generated question. The primary contribution of this work is a foundational set of benchmark datasets, designed to support and evaluate key components for a chatbot for clinical trial search."
}