Multimodal survival analysis using optimal transport matching and global-local feature fusion

Abstract

The multimodal survival analysis aims to predict patient's mortality risk using multiple modality data. Unlike unimodal task, the essence of multimodal survival prediction is to efficiently integrate the information of different modalities to make more accurate judgments. Despite recent advancements, there are still some serious problems waiting to be properly addressed, such as: (1) there are huge structural differences between gigapixel pathological images and radiological or genetic data. (2) existing multiple instance methods mainly focus on local feature representation, often neglecting global features such as tissue spatial information. In order to reasonably solve these problems, we propose a multimodal survival framework, called the optimal transport (OT) matching and global-local feature fusion (OTGL) framework. Specifically, we first use specially crafted encoders for extracting the class tokens and instance tokens from different data. Then, by applying the optimal transport to the instance tokens of different modalities, the weighted features can be obtained. After that, we perform feature fusion for class tokens and the processed local features to derive the final features that can be used to predict risk score. And we use hybrid loss function to train our OTGL and apply it to pathology-radiomic and pathology-genomic cancer datasets, many experiments demonstrate that the proposed OTGL have better performance than existing state-of-the-art methods. In practical medical settings, our model can aid clinicians in identifying high-risk patients and personalizing treatment plans. The source code will be made available at https://github.com/2018213444/OTGLmodel upon acceptance of the manuscript for publication.