@inproceedings{pal-sankarasubbu-2024-gemini,
    title = "Gemini Goes to {M}ed School: Exploring the Capabilities of Multimodal Large Language Models on Medical Challenge Problems {\&} Hallucinations",
    author = "Pal, Ankit  and
      Sankarasubbu, Malaikannan",
    editor = "Naumann, Tristan  and
      Ben Abacha, Asma  and
      Bethard, Steven  and
      Roberts, Kirk  and
      Bitterman, Danielle",
    booktitle = "Proceedings of the 6th Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.clinicalnlp-1.3/",
    doi = "10.18653/v1/2024.clinicalnlp-1.3",
    pages = "21--46",
    abstract = "Large language models have the potential to be valuable in the healthcare industry, but it{'}s crucial to verify their safety and effectiveness through rigorous evaluation. In our study, we evaluated LLMs, including Google{'}s Gemini, across various medical tasks. Despite Gemini{'}s capabilities, it underperformed compared to leading models like MedPaLM 2 and GPT-4, particularly in medical visual question answering (VQA), with a notable accuracy gap (Gemini at 61.45{\%} vs. GPT-4V at 88{\%}). Our analysis revealed that Gemini is highly susceptible to hallucinations, overconfidence, and knowledge gaps, which indicate risks if deployed uncritically. We also performed a detailed analysis by medical subject and test type, providing actionable feedback for developers and clinicians. To mitigate risks, we implemented effective prompting strategies, improving performance, and contributed to the field by releasing a Python module for medical LLM evaluation and establishing a leaderboard on Hugging Face for ongoing research and development. Python module can be found at https://github.com/promptslab/RosettaEval"
}