Multi-axis vision transformer for medical image segmentation

Abstract

Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have shown remarkable success in medical image segmentation, but individually, they struggle to capture both local and global contexts. To address this limitation, we propose MaxViT-UNet, a hybrid encoder–decoder architecture that integrates convolutional operations and multi-axis self-attention to capture local and global context for effective medical image segmentation. Our novel Hybrid Decoder fuses upsampled decoder features with encoder skip connections and refines them using a multi-axis attention block, repeated across decoding stages for progressive segmentation refinement. Experimental evaluation on the MoNuSeg18 and MoNuSAC20 datasets demonstrates that MaxViT-UNet outperforms traditional CNN-based U-Net by 2.36% and 14.14% Dice score, respectively. Similarly it outperforms Swin-UNet by 5.31% on MoNuSeg18 and nearly doubles the Dice score on MoNuSAC20. These results confirm the generalization and effective segmentation capabilities of our hybrid architecture across diverse histopathological datasets.