@inproceedings{dada-etal-2025-biomedical,
    title = "Does Biomedical Training Lead to Better Medical Performance?",
    author = "Dada, Amin  and
      Kora{\c{s}}, Osman Alperen  and
      Bauer, Marie  and
      Corbeil, Jean-Philippe  and
      Contreras, Amanda Butler  and
      Seibold, Constantin Marc  and
      Smith, Kaleb E  and
      Julian.friedrich@uk-essen.de, Julian.friedrich@uk-essen.de  and
      Kleesiek, Jens",
    editor = "Dhole, Kaustubh  and
      Clinciu, Miruna",
    booktitle = "Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM{\texttwosuperior})",
    month = jul,
    year = "2025",
    address = "Vienna, Austria and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/transition-to-people-yaml/2025.gem-1.5/",
    pages = "46--59",
    ISBN = "979-8-89176-261-9",
    abstract = "Large Language Models (LLMs) hold significant potential for improving healthcare applications, with biomedically adapted models promising enhanced performance on medical tasks. However, the effectiveness of biomedical domain adaptation for clinical tasks remains uncertain. In this study, we conduct a direct comparison of 12 biomedically adapted models and their general-domain base counterparts across six clinical tasks. Our results reveal that 11 out of 12 biomedical models exhibit performance declines, challenging prior findings that reported positive effects of biomedical adaptation. Notably, previous positive results primarily relied on multiple-choice evaluations, which may not reflect performance in real-world clinical applications. To promote reproducibility and further research, we open-source our evaluation pipeline, providing a resource for the development of models with practical benefits in healthcare settings."
}