Techniques and challenges for nuclei segmentation in cervical smear images: a review

Abstract

Cervical cancer is one of the fastest-growing cancers affecting women, leading to a significant number of deaths. However, early detection and timely treatment can greatly reduce the mortality rate and improve the chances of recovery. A widely used method for the diagnosis of cancer is the manual analysis of tissue biopsy specimens on slides. This manual process of specimen examination is time-consuming and error-prone, resulting in an increasing interest in digitizing histopathological workflows to refine and expedite analysis. This paper has compiled a comprehensive review of automated cervical nuclei segmentation approaches in histopathological images. We have examined both deep learning and traditional image segmentation methods, working mechanisms, and their variants, as well as the datasets used to evaluate these approaches. To find relevant studies, we searched on platforms such as IEEE Xplore, Google Scholar, ACM Digital Library, SpringerLink, and ScienceDirect using keywords such as cervical nuclei, nuclear, and nucleus, deep learning network (DNN), convolutional neural networks (CNNs), cervical cytology or histopathology, and traditional or classical image segmentation methods. We reviewed 78 research papers on both classical image segmentation and deep learning-based techniques specifically designed for nuclei segmentation in cervical histopathological images, published from 2010 until October 2024. We organized these studies into two main categories: Classical image segmentation methods and deep learning approaches, subdividing them into relevant subcategories and compiled a comparative analysis of their results, identified ongoing challenges in nuclei segmentation, and highlighted opportunities and future prospects for this task. We discussed recent studies on cervical nuclei segmentation using automated methods. The implementation of automated image segmentation methods and their various extensions has significantly improved the performance of automated diagnostic systems for cervical cancer.