@inproceedings{corbeil-etal-2026-medriskeval,
    title = "{M}ed{R}isk{E}val: Medical Risk Evaluation Benchmark of Language Models, On the Importance of User Perspectives in Healthcare Settings",
    author = "Corbeil, Jean-Philippe  and
      Kim, Minseon  and
      Griot, Maxime  and
      Agarwal, Sheela  and
      Sordoni, Alessandro  and
      Beaulieu, Francois  and
      Vozila, Paul",
    editor = {Matusevych, Yevgen  and
      Eryi{\u{g}}it, G{\"u}l{\c{s}}en  and
      Aletras, Nikolaos},
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 5: Industry Track)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-industry.39/",
    pages = "513--524",
    ISBN = "979-8-89176-384-5",
    abstract = "As the performance of large language models (LLMs) continues to advance, their adoption in the medical domain is increasing. However, most existing risk evaluations largely focused on general safety benchmarks. In the medical applications, LLMs may be used by a wide range of users, ranging from general users and patients to clinicians, with diverse levels of expertise and the model{'}s outputs can have a direct impact on human health which raises serious safety concerns. In this paper, we introduce MedRiskEval, a medical risk evaluation benchmark tailored to the medical domain. To fill the gap in previous benchmarks that only focused on the clinician perspective, we introduce a new patient-oriented dataset called PatientSafetyBench containing 466 samples across 5 critical risk categories. Leveraging our new benchmark alongside existing datasets, we evaluate a variety of open- and closed-source LLMs. To the best of our knowledge, this work establishes an initial foundation for safer deployment of LLMs in healthcare."
}