Understanding and tackling the modality imbalance problem in multimodal survival prediction

Benefiting from the in-depth integration of multimodal data, survival prediction has emerged as a pivotal task in cancer prognosis by facilitating personalized treatment planning and medical resource allocation. In this study, we report an intriguing phenomenon of inter-modality capability gap (ICG) enlargement during joint survival modelling of genomics data and pathology images. This observation, supported by our dedicated theoretical analysis, uncovers a previously unrecognized modality imbalance problem, where pathology modality suffers from limited gradient propagation and insufficient learning while genomics modality dominates in reducing survival loss. To tackle this problem, we further propose a balanced multimodal learning approach for survival prediction named BMLSurv, which introduces two innovative auxiliary learning strategies: self-enhancement learning (SEL) and peer-assistance learning (PAL). The SEL strategy exploits a real-time imbalance measure to guide extra task-aware supervision, therefore dynamically strengthening pathology-specific gradient propagation in a self-enhanced manner. Meanwhile, the PAL strategy leverages the stronger genomics modality as a “helpful peer” to assist the sufficient learning of pathology modality via a new risk-ranking distillation technique. Extensive experiments on representative cancer datasets demonstrate that by successfully address the modality imbalance problem, BMLSurv remarkably narrows the ICG in joint survival modelling and consistently outperforms state-of-the-art methods by a large margin. These results underscore the potential of BMLSurv to advance multimodal survival prediction and enhance clinical decision-making in cancer prognosis.