Enhancing multimodal medical image analysis with Slice-Fusion: A novel fusion approach to address modality imbalance

Abstract

Background and objective:

In recent times, medical imaging analysis (MIA) has seen an increasing interest due to its core application in computer-aided diagnosis systems (CADs). A modality in MIA refers to a specific technology used to produce human body images, such as MRI, CT scans, or X-rays. Each modality presents unique challenges and characteristics, often leading to imbalances within datasets. This significant challenge impedes model training and generalization due to the varying convergence rates of different modalities and the suppression of gradients in less dominant modalities.

Methods:

This paper proposes a novel fusion approach, and we named it Slice-Fusion. The proposed approach aims to mitigate the modality imbalance problem by implementing a “Modality-Specific-Balancing-Factor” fusion strategy. Furthermore, it incorporates an auxiliary (uni-modal) task that generates balanced modality pairs based on the image orientations of different modalities. Subsequently, a novel multimodal classification framework is presented to learn from the generated balanced modalities. The effectiveness of the proposed approach is evaluated through comparative assessments on a publicly available BraTS2021 dataset. The results demonstrate the efficiency of Slice-Fusion in resolving the modality imbalance problem. By enhancing the representation of balanced features and reducing modality bias, this approach holds promise for advancing visual health informatics and facilitating more accurate and reliable medical image analysis.

Results:

In the experiment section, three diverse experiments are conducted such as i) Fusion Loss Metrics Evaluation, ii) Classification, and iii) Visual Health Informatics. Notably, the proposed approach achieved an F1-Score of (100%, 81.25%) on the training and validation sets for the classification generalization task. In addition to the Slice-Fusion’s out-performance, the study also created a new modality-aligned dataset (a highly balanced and informative modality-specific image collection) that aids further research and improves MIA’s robustness. These advancements not only enhance the capability of medical diagnostic tools but also create opportunities for future innovations in the field.

Conclusion:

This study contributes to advancing medical image analysis, such as effective modality fusion, image reconstruction, comparison, and glioma classification, facilitating more accurate and reliable results, and holds promise for further advancements in visual health informatics.