Multi-modal Intermediate Fusion Model for diagnosis prediction

Abstract

The goal of the diagnostic prediction task is to predict what disease patients are likely to have at their next visit, based on their historical electronic medical records. Existing studies mainly conduct the prediction task by separately using discrete medical codes or clinical notes. However, few existing studies fuse multi-modal features from medical codes and clinical notes together for diagnostic prediction. Practically, using multiple modes of EHRs data can obtain more complete patient representation to improve the predictive performance of the model. Therefore, we proposed a Multi-modal intermediate Fusion Model (MFM) to predict patient diagnosis based on diagnostic codes and clinical notes. Specifically, MFM is mainly based on recurrent neural network to model data in different modes to extract effective features. Then, an intermediate fusion module is used to not only extract the global context information of data in each mode, but also capture the correlation between data in different modes. Finally, a multi-modal fusion matrix is generated for diagnosis prediction. Experimental results on a real dataset show that the proposed method improves the prediction performance compared with the baseline methods.