ChromSeg-P3GAN: A Benchmark Dataset and Pix2Pix Patch Generative Adversarial Network for Chromosome Segmentation

Chromosome image analysis with automated karyotyping systems (AKS) is crucial for the diagnosis and prognosis of hematologic malignancies and genetic disorders. However, the partial or complete occlusion of nonrigid chromosome structures significantly limits the performance of AKS. To address these challenges, this paper extends the Pix2Pix generative adversarial network (GAN) model for the first time to segment overlapping and touching chromosomes. A new publicly available dataset of G-banded metaphase chromosome images has been prepared specifically for this study, marking the first use of GAN-based methods on such data, as previous research has been confined to FISH image datasets. A comprehensive comparative study of Pix2Pix GAN objective functions—including binary cross entropy (BCE) loss with and without logit, Tversky loss, Focal Tversky (FT) loss with different gamma values, and Dice loss—has been conducted. To address class imbalance and segmentation challenges, a custom loss function combining BCE with logit, Tversky loss, and L1 loss is introduced, which yields superior performance. Furthermore, a 5-fold cross-validation is performed to evaluate the stability and performance of the models. The top five models from the comparative study are tested on a completely unseen dataset, and their performance is visualized using a boxplot. The proposed model demonstrates the best segmentation performance, with Intersection over Union (IoU) of 0.9247, Dice coefficient of 0.9596, and recall of 0.9687. The results validate the robustness and effectiveness of the proposed approach for addressing overlapping and touching chromosome segmentation in AKS.