A Semi-Interpretable Multimodal Model Combining Electronic Health Records and Chest X-Rays for COVID-19-Related Death Prediction

Abstract

Purpose

Prognostic models can benefit from integrating heterogeneous data sources. While deep learning-based multimodal models often improve predictive performance, their limited interpretability remains a barrier to clinical adoption. Moreover, achieving robust generalization with such models typically requires large and diverse datasets, which are not always available in clinical research. This study evaluated whether a late fusion model combining traditional statistical approaches with deep learning models could rival deep learning-only multimodal models while preserving interpretability to support clinical use.

Methods

We conducted a retrospective single-center cohort study of 965 hospitalized patients with laboratory-confirmed SARS-CoV-2 infection who underwent frontal chest X-ray imaging during hospitalization between March 2020 and January 2022. Of these, 64 (6.6%) died. Structured clinical data and chest X-ray images were used to predict COVID-19-related death. We developed unimodal baseline models and two multimodal approaches: a deep learning-based joint fusion model and a late fusion model leveraging logistic regression to combine predictions from the best-performing models for every data modality. Given the limited sample size and class imbalance, performance was evaluated using repeated five-fold cross validation.

Results

The late fusion model achieved the best overall performance, with the highest median test AUPRC of 0.363 (IQR 0.271–0.384) across cross validation folds, outperforming unimodal baselines based on structured clinical data or images alone. The late fusion model also outperformed the joint fusion model, achieving, respectively, a 3.1% and 3.6% relative improvement in AUPRC and Brier score on pooled out-of-sample predictions and yielding the highest balanced F1-score. Moreover, it preserved partial interpretability by linking predictions to clinically meaningful variables such as age and comorbidities and enabling image-based predictions to be traced back to radiographic abnormalities.

Conclusions

Late fusion models integrating statistical models for clinical data with deep learning for image classification can enhance prognostic performance over unimodal models without fully sacrificing interpretability. In settings with limited data, they may also outperform deep learning-based joint fusion models. These findings support late fusion as a promising strategy that warrants prospective evaluation and external validation before clinical implementation.