HDPL: Hypergraph-Based Dynamic Prompting Learning for Incomplete Multimodal Medical Learning

Abstract

Multimodal learning has garnered significant attention in the medical field due to its ability to provide a more comprehensive perspective utilizing various types of data, that aids in making more accurate decisions. However, the complexity of medical data, coupled with missing modalities, severely hinders predictive accuracy. Existing methods for multimodal learning with missing modalities still face considerable challenges. For instance, approaches that construct multimodal shared feature spaces often result in high computational costs, while methods that infer missing modalities based on complete ones may overly rely on the complete modalities, potentially skewing results. Pre-trained transformer methods address these issues but still have limitations, such as it can only process one missing modality at testing-stage. This is partly because structured data, unlike sequential data, lacks inherent minimum semantic units or natural order. Additionally, the positional encodings generated by this type of methods may introduce information interference when applied to structured data, leading to poor alignment with sequential data during modality fusion in transformer models. To tackle these challenges, we introduce HDPL: Hypergraph-based Dynamic Prompt Learning for Incomplete Multimodal Medical Learning, comprising three modules. The High-Order Hypergraph Embedding module can identify the minimal semantic units within structured data and utilizes hypergraph structures to extract high-dimensional features from clinical data. The Multimodal Medical Data Integrator module closes the distance of the embedding vectors corresponding in the shared space of modality-features, facilitating the integration of modalities in transformer. The Dynamic Network Structure Optimization module is a dynamic learning network by dynamically change the width and depth of network, improving the overall performance of the model, and it alleviates the shortcomings caused by incomplete modality to some extent. Through comprehensive experimentation, we demonstrate the efficiency and robustness of our model in dealing missing modalities and reducing training-burdens.