Tumor Image Segmentation Using Multimodal Generation and Vision Transformer Integration

Abstract

Accurate tumor segmentation is crucial for clinical diagnosis and radiotherapy planning. However, computed tomography (CT)-only or magnetic resonance imaging (MRI)-only imaging often fails to capture the full tumor morphology due to modality-specific limitations. To address these limitations and leverage the complementary strengths of CT and MRI, we propose multimodal generative polygon-vision transformer (MGP-ViT), which integrates CT-to-MRI generative synthesis with a polygon-based boundary representation to enhance tumor boundary fidelity. Specifically, a cycle-consistent adversarial network (CycleGAN)-based CT-to-MRI translator generates synthetic MRI scans from unpaired CT data, improving soft-tissue contrast without requiring spatially paired CT–MRI training samples. These synthetic MRI and original CT scans are input to a dual-branch vision transformer (ViT) encoder: each branch extracts modality-specific features, which are fused via a modality-aware cross-attention module and then decoded into polygon vertex sequences that outline tumor regions. We further introduce a convolutional smoothing loss (CSL) that encourages locally smooth vertex trajectories by penalizing large, incoherent jumps while tolerating small spatial offsets. Experiments on the public MSD pancreas dataset and a private cervical-cancer CT cohort show that MGP-ViT outperforms state-of-the-art methods in terms of intersection over union (IoU) and dice, while offering improved boundary precision, greater robustness to imaging variability, and better cross-dataset generalization.