Abstract
The MEDIQA-M3G 2024 challenge necessitates novel solutions for Multilingual & Multimodal Medical Answer Generation in dermatology (wai Yim et al., 2024a). This paper addresses the limitations of traditional methods by proposing a weakly supervised learning approach for open-ended medical question-answering (QA). Our system leverages readily available MEDIQA-M3G images via a VGG16-CNN-SVM model, enabling multilingual (English, Chinese, Spanish) learning of informative skin condition representations. Using pre-trained QA models, we further bridge the gap between visual and textual information through multimodal fusion. This approach tackles complex, open-ended questions even without predefined answer choices. We empower the generation of comprehensive answers by feeding the ViT-CLIP model with multiple responses alongside images. This work advances medical QA research, paving the way for clinical decision support systems and ultimately improving healthcare delivery.- Anthology ID:
- 2024.clinicalnlp-1.31
- Volume:
- Proceedings of the 6th Clinical Natural Language Processing Workshop
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Tristan Naumann, Asma Ben Abacha, Steven Bethard, Kirk Roberts, Danielle Bitterman
- Venues:
- ClinicalNLP | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 339–345
- Language:
- URL:
- https://aclanthology.org/2024.clinicalnlp-1.31
- DOI:
- Cite (ACL):
- Nadia Saeed. 2024. MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 339–345, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning (Saeed, ClinicalNLP-WS 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.clinicalnlp-1.31.pdf