Evaluation of general-purpose large language models as diagnostic support tools in cervical cytology.

Introduction: The application of general-purpose large language models (LLMs) in cytopathology remains largely unexplored. This study aims to evaluate the accuracy and consistency of a custom version of ChatGPT-4 (GPT), ChatGPT o3, and Gemini 2.5 Pro as diagnostic support tools for cervical cytology.

Materials and methods: A total of 200 Papanicolaou-stained cervical cytology images were acquired at 40x magnification, each measuring 384 × 384 pixels. These images consisted of 100 cases classified as negative for intraepithelial lesion or malignancy (NILM) and 100 cases across various abnormal categories: 20 low-grade squamous intraepithelial lesion (LSIL), 20 high-grade squamous intraepithelial lesion (HSIL), 20 squamous cell carcinoma (SCC), 20 adenocarcinoma in situ (AIS), and 20 adenocarcinoma (ADC). Diagnostic accuracy and consistency were evaluated by submitting each image to a GPT, ChatGPT o3, and Gemini 2.5 Pro 5-10 times.

Results: When distinguishing normal from abnormal cytology, LLMs showed mean sensitivity between 85.4 % and 100 %, and specificity between 67.2 % and 92.7 %. ChatGPT o3 was more accurate in identifying NILM (mean 89.2 % vs. 67.2 %) but less accurate in detecting LSIL (34 % vs. 85 %), HSIL (6 % vs. 63 %), and ADC (28 % vs. 91 %). Chain-of-thought prompting and submitting multiple images of the same diagnosis to ChatGPT o3 and Gemini 2.5 Pro did not significantly improve accuracy. Both models also performed poorly in identifying cervicovaginal infections.

Conclusions: ChatGPT o3 and Gemini 2.5 Pro demonstrated complementary strengths in cervical cytology. Due to their low accuracy and inconsistency in abnormal cytology, general-purpose LLMs are not recommended as diagnostic support tools in cervical cytology.