An edge enhanced 3D mamba U-Net for pediatric brain tumor segmentation with transfer learning.

Abstract

Background: Pediatric gliomas, particularly high-grade subtypes, are highly aggressive tumors with low survival rates, and their segmentation remains challenging due to distinct imaging characteristics and data scarcity. While deep learning models perform well in adult glioma segmentation, they struggle with pediatric gliomas, particularly in segmenting complex regions such as the tumor core (TC) and enhancing tumor (ET).

Purpose: This study proposes a solution to address the dual challenges of complex tumor morphology and limited pediatric data in MRI-based pediatric brain tumor segmentation.

Methods: A solution utilizing an edge-enhanced 3D Mamba U-Net model combined with transfer learning was proposed for pediatric brain tumor segmentation. The network integrated U-Net's multi-scale feature extraction with Mamba's global dependency modeling, augmented by a Mamba residual (MR) block. An edge enhancement (EE) module was embedded in the skip-connection layers to refine boundary detection and capture local features in small pediatric tumor regions. Finally, a non-encoder fine-tuning (NEF) strategy was adopted to adapt the pre-trained adult model to pediatric data by updating only the final reconstruction stage while preserving learned representations. The model was pre-trained on the BraTS 2021 dataset (1251 adult glioma training cases) and fine-tuned on the BraTS-PEDs 2023 dataset (99 pediatric glioma training cases, split 7:1:2 for training, validation, and testing).

Results: On the BraTS-PEDs 2023 dataset, the method achieved average Dice scores of 0.8917 (WT), 0.8557 (TC), and 0.6365 (ET), with corresponding Hausdorff distances of 3.82, 5.14, and 3.53. The proposed method outperformed the baseline and existing pediatric glioma segmentation approaches included in our experiments.

Conclusions: The 3D Mamba U-Net with transfer learning and edge-enhancement modules effectively alleviates the challenges of complex tumor boundaries and small sample size problem in pediatric glioma segmentation.