@inproceedings{xu-staniek-2025-multimodal,
    title = "Multimodal Transformers for Clinical Time Series Forecasting and Early Sepsis Prediction",
    author = "Xu, Jinghua  and
      Staniek, Michael",
    editor = "Ananiadou, Sophia  and
      Demner-Fushman, Dina  and
      Gupta, Deepak  and
      Thompson, Paul",
    booktitle = "Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Health)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.cl4health-1.8/",
    pages = "100--108",
    ISBN = "979-8-89176-238-1",
    abstract = "Sepsis is a leading cause of death in Intensive Care Units (ICU). Early detection of sepsis is crucial to patient survival. Existing works in the clinical domain focus mainly on directly predicting a ground truth label that is the outcome of a medical syndrome or condition such as sepsis. In this work, we primarily focus on clinical time series forecasting as a means to solve downstream predictive tasks intermediately. We base our work on a strong monomodal baseline and propose multimodal transformers using set functions via fusing both physiological features and texts in electronic health record (EHR) data. Furthermore, we propose hierarchical transformers to effectively represent clinical document time series via attention mechanism and continuous time encoding. Our multimodal models significantly outperform baseline on MIMIC-III data by notable gaps. Our ablation analysis show that our atomic approaches to multimodal fusion and hierarchical transformers for document series embedding are effective in forecasting. We further fine-tune the forecasting models with labelled data and found some of the multimodal models consistently outperforming baseline on downstream sepsis prediction task."
}