Automatic cancer nuclei segmentation on histological images: comparison study of deep learning methods

Cancer is one of the most common health problems affecting individuals worldwide. In the field of biomedical engineering, one of the main methods for cancer diagnosis is the analysis of histological images of tissue structures and cell nuclei using artificial intelligence. Here, we compared the performance of 15 deep learning methods viz: UNet, Deep-UNet, UNet-CBAM, RA-UNet, SA-Unet and Nuclei-SegNet, UNet-VGG2016, UNet-Resnet-101, TransResUNet, Inception-UNet, Att-UNet++ , FF-UNet, Att-UNet, Res-UNet and a new model, DanNucNet, in pathological nuclei segmentation on tissue slices from different organs on five open datasets: MoNuSeg, CoNSeP, CryoNuSeg, Data Science Bowl, and NuInsSeg. Before training on the data, the pixel intensity and color distribution were analyzed, and different augmentation techniques were applied. The results showed that the UNet-based model with 34.57 million Deep-UNet parameters performed the best, outperforming all models in terms of the Dice coefficient from 3.13 to 22.91%. The implementation of Deep-UNet in this context provides a valuable tool for accurate extraction of cancer cell nuclei from histological images, which in turn will contribute to further developments in cancer pathology and digital histology.