Multimodal representation learning with hierarchical knowledge decomposition for cancer survival analysis

Abstract

The accurate survival analysis of cancer patients serves as an important decision-making basis for formulating personalized treatment plans. Recent studies illustrate that integrating multimodal information including clinical diagnostic data, genomic features, and whole-slide images (WSIs) can significantly enhance the performance of prognostic assessment models. However, most existing multimodal survival analysis methods excessively focus on extracting shared inter-modal features while failing to effectively mine modality-specific biological information, resulting in inadequate acquisition of comprehensive patient multimodal representations. Furthermore, how to extract prognosis-related tissue microenvironment features from ultra-high-resolution WSIs remains an unresolved open challenge. To address these issues, a multimodal representation learning framework with hierarchical knowledge decomposition (MRL-HKD) is proposed for cancer survival analysis. MRL-HKD transforms multimodal representation learning into a set partitioning problem within multimodal knowledge spaces, and employs a hierarchical multimodal knowledge decomposition module to decouple complex inter-modal relationships. Meanwhile, to address the challenge of high-dimensional pathological image feature extraction, a gated attention mechanism-based multimodal patch attention network is designed. The performance comparison experiments on four cancer datasets demonstrate that MRL-HKD significantly outperforms state-of-the-art methods. Our study demonstrates the potential of gated attention mechanisms and hierarchical knowledge decomposition in multimodal survival analysis, and provides an effective tool for cancer prognosis prediction. The source code will be open-sourced at https://github.com/yangxulin/MRL-HKD.