3MU-Net: A multi-layer, multi-view and multi-modal segmentation model for PET/CT images of lung tumors

Abstract

Multimodal medical images provide reliable and accurate information on the lesion categories, which is essential for diagnosing lung cancer. In medical images, the adequate extraction of lesion cross-modal features and lesion contextual semantic information is a key issue. 3MU-Net is proposed by this paper, which is a Multi-layer, Multi-view and Multi-modal lung tumors segmentation model. The main contributions of 3MU-Net are as follows: Firstly, CMformer uses parallel CNN branch and Transformer branch to learn cross-modal medical image features, which builds cross-modal pixel dependencies between global features and local features. Secondly, Cross-view Context-aware Processor is designed, there are four Deep-Shallow Feature Enhancement Modules and a Cross-view Attention Module in Cross-view Context-aware Processor. Deep-Shallow Feature Enhancement Module uses a bidirectional learning method to guide the information interaction between shallow-level features and deep-level features, which reduces the semantic information gap between adjacent layers. In addition, the Deep-Shallow Feature Enhancement Module aggregates coarse-grained semantic features and fine-grained detailed features. Cross-view Attention Module is used to obtain multi-scale lesion information through multi-view feature extraction, which extends the diversity of spatial features in the lesion area. Finally, the effectiveness of 3MU-Net is validated on a clinical multimodal lung medical image dataset. The results for Miou, Dice, Voe, Rvd, Acc, and Recall are 90.97%, 95.23%, 93.87%, 94.13%, 97.97%, and 94.45%, respectively. It is great significance for lung tumors segmentation. Code is available at: https://github.com/xiaoniu1030/ 3MU-Net.