@inproceedings{arias-duart-etal-2025-automatic,
    title = "Automatic Evaluation of Healthcare {LLM}s Beyond Question-Answering",
    author = "Arias-Duart, Anna  and
      Martin-Torres, Pablo Agustin  and
      Hinjos, Daniel  and
      Bernabeu-Perez, Pablo  and
      Ganzabal, Lucia Urcelay  and
      Mallo, Marta Gonzalez  and
      Gururajan, Ashwin Kumar  and
      Lopez-Cuena, Enrique  and
      Alvarez-Napagao, Sergio  and
      Garcia-Gasulla, Dario",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/moar-dois/2025.naacl-short.10/",
    doi = "10.18653/v1/2025.naacl-short.10",
    pages = "108--130",
    ISBN = "979-8-89176-190-2",
    abstract = "Current Large Language Models (LLMs) benchmarks are often based on open-ended or close-ended QA evaluations, avoiding the requirement of human labor. Close-ended measurements evaluate the factuality of responses but lack expressiveness. Open-ended capture the model{'}s capacity to produce discourse responses but are harder to assess for correctness. These two approaches are commonly used, either independently or together, though their relationship remains poorly understood. This work is focused on the healthcare domain, where both factuality and discourse matter greatly. It introduces a comprehensive, multi-axis suite for healthcare LLM evaluation, exploring correlations between open and close benchmarks and metrics. Findings include blind spots and overlaps in current methodologies. As an updated sanity check, we release a new medical benchmark{--}CareQA{--}, with both open and closed variants. Finally, we propose a novel metric for open-ended evaluations {--}Relaxed Perplexity{--} to mitigate the identified limitations."
}