WaveSeg‐UNet model for overlapped nuclei segmentation from multi‐organ histopathology images

Nuclei segmentation is a challenging task in histopathology images. It is challenging due to the small size of objects, low contrast, touching boundaries, and complex structure of nuclei. Their segmentation and counting play an important role in cancer identification and its grading. In this study, WaveSeg‐UNet, a lightweight model, is introduced to segment cancerous nuclei having touching boundaries. Residual blocks are used for feature extraction. Only one feature extractor block is used in each level of the encoder and decoder. Normally, images degrade quality and lose important information during down‐sampling. To overcome this loss, discrete wavelet transform (DWT) alongside max‐pooling is used in the down‐sampling process. Inverse DWT is used to regenerate original images during up‐sampling. In the bottleneck of the proposed model, atrous spatial channel pyramid pooling (ASCPP) is used to extract effective high‐level features. The ASCPP is the modified pyramid pooling having atrous layers to increase the area of the receptive field. Spatial and channel‐based attention are used to focus on the location and class of the identified objects. Finally, watershed transform is used as a post processing technique to identify and refine touching boundaries of nuclei. Nuclei are identified and counted to facilitate pathologists. The same domain of transfer learning is used to retrain the model for domain adaptability. Results of the proposed model are compared with state‐of‐the‐art models, and it outperformed the existing studies.