Prompt-guided orthogonal multimodal fusion for cancer survival prediction

Abstract

Cancer survival prediction can assist clinicians in developing personalized treatment plans for patients. Comprehensive cancer diagnosis and treatment require integrating macroscopic and microscopic imaging. However, significant discrepancies in the spatial resolution and anatomical scale between imaging modalities hinder existing multimodal fusion methods from effectively learning correlated semantic features with limited datasets. In this work, we introduce a prompt-guided orthogonal multimodal fusion strategy (POMF) for fusing multimodal medical images across anatomical scales. POMF utilizes modality-specific prompts to fine-tune pretrained models, facilitating bias adaptation to medical imaging features while ensuring more computationally efficient training. A modality consistency-discrepancy prototype is designed as the modality-inherent prompt in POMF, disentangling the multimodal features and bridging the potential correlations across the orthogonal modalities. POMF is validated on a glioma survival prediction task using paired radiology and pathology images. The experiment results suggest that POMF achieves superior C-index with existing full-tuning and prompt-tuning methods. Additionally, the ablation studies demonstrate that POMF is adaptable to various architectures of pretrained encoders and multiple multimodal fusion strategies on cross-scale medical images.