@inproceedings{saeed-2024-medifact,
    title = "{M}edi{F}act at {MEDIQA}-{M}3{G} 2024: Medical Question Answering in Dermatology with Multimodal Learning",
    author = "Saeed, Nadia",
    editor = "Naumann, Tristan  and
      Ben Abacha, Asma  and
      Bethard, Steven  and
      Roberts, Kirk  and
      Bitterman, Danielle",
    booktitle = "Proceedings of the 6th Clinical Natural Language Processing Workshop",
    month = jun,
    year = "2024",
    address = "Mexico City, Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.clinicalnlp-1.31/",
    doi = "10.18653/v1/2024.clinicalnlp-1.31",
    pages = "339--345",
    abstract = "The MEDIQA-M3G 2024 challenge necessitates novel solutions for Multilingual {\&} Multimodal Medical Answer Generation in dermatology (wai Yim et al., 2024a). This paper addresses the limitations of traditional methods by proposing a weakly supervised learning approach for open-ended medical question-answering (QA). Our system leverages readily available MEDIQA-M3G images via a VGG16-CNN-SVM model, enabling multilingual (English, Chinese, Spanish) learning of informative skin condition representations. Using pre-trained QA models, we further bridge the gap between visual and textual information through multimodal fusion. This approach tackles complex, open-ended questions even without predefined answer choices. We empower the generation of comprehensive answers by feeding the ViT-CLIP model with multiple responses alongside images. This work advances medical QA research, paving the way for clinical decision support systems and ultimately improving healthcare delivery."
}