Towards robust multimodal ultrasound classification for liver tumor diagnosis: A generative approach to modality missingness

Background and Objective

In medical image analysis, combining multiple imaging modalities enhances diagnostic accuracy by providing complementary information. However, missing modalities are common in clinical settings, limiting the effectiveness of multimodal models. This study addresses the challenge of missing modalities in liver tumor diagnosis by proposing a generative model-based method for cross-modality reconstruction and classification. The dataset for this study comprises 359 case data from a hospital, with each case including three modality data: B-mode ultrasound images, Color Doppler Flow Imaging (CDFI), and clinical data. Only cases with one missing image modality are considered, excluding those with missing clinical data.

Methods

We developed a multimodal classification framework specifically for liver tumor diagnosis, employing various feature extraction networks to explore the impact of different modality combinations on classification performance when only available modalities are used. DenseNet extracts CDFI features, while EfficientNet is employed for B-mode ultrasound image feature extraction. These features are then flattened and concatenated with clinical data using feature-level fusion to obtain a full-modality model. Modality weight parameters are introduced to emphasize the importance of different modalities, yielding Model_D, which serves as the classification model after subsequent image modality supplementation. In cases of missing modalities, generative models, including U-GAT-IT and MSA-GAN, are utilized for cross-modal reconstruction of missing B-mode ultrasound or CDFI images (e.g., reconstructing CDFI from B-mode ultrasound when CDFI is missing). After evaluating the usability of the generated images, they are input into Model_D as supplementary images for the missing modalities.

Results

Model performance and modality supplementation effects were evaluated through accuracy, precision, recall, F1 score, and AUC metrics. The results demonstrate that the proposed Model_D, which introduces modality weights, achieves an accuracy of 88.57 %, precision of 87.97 %, recall of 82.32 %, F1 score of 0.87, and AUC of 0.95 in the full-modality classification task for liver tumors. Moreover, images reconstructed using U-GAT-IT and MSA-GAN across modalities exhibit PSNR > 20 and multi-scale structural similarity > 0.7, indicating moderate image quality with well-preserved overall structures, suitable for input into the model as supplementary images in cases of missing modalities. The supplementary CDFI or B-mode ultrasound images achieve 87.10 % and 86.43 % accuracy, respectively, with AUC values of 0.92 and 0.95. This proves that even in the absence of certain modalities, the generative models can effectively reconstruct missing images, maintaining high classification performance comparable to that in complete modality scenarios.

Conclusions

The generative model-based approach for modality reconstruction significantly improves the robustness of multimodal classification models, particularly in the context of liver tumor diagnosis. This method enhances the clinical applicability of multimodal models by ensuring high diagnostic accuracy despite missing modalities. Future work will explore further improvements in modality reconstruction techniques to increase the generalization and reliability of the model in various clinical settings.