Advancing MRI segmentation with CLIP-driven semi-supervised learning and semantic alignment

Abstract

Precise segmentation and reconstruction of multi-structures within MRI are crucial for clinical applications such as surgical navigation. However, medical image segmentation faces several challenges. Although semi-supervised methods can reduce the annotation workload, they often suffer from limited robustness. To address this issue, this study proposes a novel CLIP-driven semi-supervised model, that includes two branches and a module. In the image branch, copy-paste is used as data augmentation method to enhance consistency learning. In the text branch, patient-level information is encoded via CLIP to drive the image branch. Notably, a novel cross-modal fusion module is designed to enhance the alignment and representation of text and image. Additionally, a semantic spatial alignment module is introduced to register segmentation results from different axial MRIs into a unified space. Three multi-modal datasets (one private and two public) were constructed to demonstrate the model’s performance. Compared to previous state-of-the-art methods, this model shows a significant advantage with both 5% and 10% labeled data. This study constructs a robust semi-supervised medical segmentation model, particularly effective in addressing label inconsistency and abnormal organ deformations. It also tackles the axial non-orthogonality challenges inherent in MRI, providing a consistent view of multi-structures.