Diagnostic performance of artificial intelligence models trained on scattered single-cell images is not preserved for hyperchromatic crowded cell groups in cervical cytology

Abstract

Background

Deep learning has shown promising performance in cervical cytology; however, many studies have relied on presegmented single-cell images rather than the more complex morphologic patterns encountered in routine practice. Here, scattered cells were defined as isolated or dissociated, nonoverlapping single cells. This study quantified the performance loss when artificial intelligence (AI) models trained on these cells were applied to hyperchromatic crowded cell groups (HCGs).

Methods

Binary convolutional neural network models were developed to differentiate between negative for intraepithelial lesion or malignancy cases and high-grade squamous intraepithelial lesion cases via a scattered cell data set composed of institutional and public liquid-based cytology images. The scattered cell data set comprised 101 cases, with 1062 images; the independent HCG data set comprised 48 cases, with 330 images. ResNet-50, ResNeXt-50, ConvNeXt-Tiny, EfficientNet-B3, VGG-19, and GoogLeNet were trained on scattered cell images, and then directly applied to HCGs without retraining or threshold recalibration.

Results

All models showed high performance on the scattered cell data set, with the area under the receiver operating characteristic curve (AUC) ranging from 0.950 to 0.996. When directly applied to HCGs, performance declined across all architectures, with the AUC ranging from 0.385 to 0.683. ConvNeXt-Tiny showed the highest AUC on HCGs (0.683); however, this remained substantially lower than its performance on scattered cells (0.996). For all architectures, the AUC was significantly lower on HCGs than on the scattered cell data set.

Conclusions

Binary AI models trained on scattered cell images achieved excellent discrimination in the original setting but their performance was not preserved when directly applied to HCGs. These findings underscore the need for direct validation and HCG-aware model design in cervical cytology AI.