Permutation invariant self-attention infused U-shaped transformer for medical image segmentation

Abstract

The size and shape of organs in the human body vary according to factors like genetics, body size, proportions, health, lifestyle, gender, ethnicity, and race. Further, abnormalities due to cancer and chronic diseases also affect the size of organs and tumors. Moreover, the spatial location and area of these organs deviates along the transverse plane (Z plane) of the medical scans. Therefore, the generalizability and robustness of a computer vision framework over medical images can be improved if the framework is also encouraged to learn representations of the target areas regardless of their spatial location in input images. Hence, we propose a novel permutation invariant multi-headed self-attention (PISA) module to reduce a U-shaped transformer-based architecture Swin-UNet’s sensitivity towards permutation. We have infused this module in the skip connection of our architecture. We have achieved a mean dice score of 79.25 on the segmentations of 8 abdominal organs, better than most state-of-the-art algorithms. Moreover, we have analyzed the generalizability of our architecture over publicly available multi-sequence cardiac MRI datasets. When tested over a sequence unseen by the model during training, 25.1 % and 9.0 % improvement in dice scores were observed in comparison to the pure-CNN-based algorithm and pure transformer-based architecture, respectively, thereby demonstrating its versatility. Replacing the Self Attention module in a U-shaped transformer architecture with our Permutation Invariant Self Attention module produced noteworthy segmentations over shuffled test images, even though the module was trained solely on normal images. The results demonstrate the enhanced efficiency of the proposed module in imparting attention to target organs irrespective of their spatial positions.