Abstract
Clinical data often exists in different forms across the lifetime of a patient’s interaction with the healthcare system - structured, unstructured or semi-structured data in the form of laboratory readings, clinical notes, diagnostic codes, imaging and audio data of various kinds, and other observational data. Formulating a representation model that aggregates information from these heterogeneous sources may allow us to jointly model on data with more predictive signal than noise and help inform our model with useful constraints learned from better data. Multimodal fusion approaches help produce representations combined from heterogeneous modalities, which can be used for clinical prediction tasks. Representations produced through different fusion techniques require different training strategies. We investigate the advantage of adding narrative clinical text to structured modalities to classification tasks in the clinical domain. We show that while there is a competitive advantage in combined representations of clinical data, the approach can be helped by training guidance customized to each modality. We show empirical results across binary/multiclass settings, single/multitask settings and unified/multimodal learning rate settings for early and late information fusion of clinical data.- Anthology ID:
- 2022.louhi-1.18
- Volume:
- Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Alberto Lavelli, Eben Holderness, Antonio Jimeno Yepes, Anne-Lyse Minard, James Pustejovsky, Fabio Rinaldi
- Venue:
- Louhi
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 154–159
- Language:
- URL:
- https://aclanthology.org/2022.louhi-1.18
- DOI:
- 10.18653/v1/2022.louhi-1.18
- Cite (ACL):
- Sneha Jha, Erik Mayer, and Mauricio Barahona. 2022. Improving information fusion on multimodal clinical data in classification settings. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 154–159, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Improving information fusion on multimodal clinical data in classification settings (Jha et al., Louhi 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.louhi-1.18.pdf