Adapting Segment Anything Model for 3D Brain Tumor Segmentation With Missing Modalities

Abstract

The problem of missing or unavailable magnetic resonance imaging modalities challenges clinical diagnosis and medical image analysis technology. Although the development of deep learning and the proposal of large models have improved medical analytics, this problem still needs to be better resolved.The purpose of this study was to efficiently adapt the Segment Anything Model, a two-dimensional visual foundation model trained on natural images, to address the challenge of brain tumor segmentation with missing modalities. We designed a twin network structure that processes missing and intact magnetic resonance imaging (MRI) modalities separately using shared parameters. It involved comparing the features of two network branches to minimize differences between the feature maps derived from them. We added a multimodal adapter before the image encoder and a spatial–depth adapter before the mask decoder to fine-tune the Segment Anything Model for brain tumor segmentation. The proposed method was evaluated using datasets provided by the MICCAI BraTS2021 Challenge. In terms of accuracy and robustness, the proposed method is better than existing solutions. The proposed method can segment brain tumors well under the missing modality condition.