Determination of an optimal quality control method for Pap test analysis using digital cytology and artificial intelligence

IF 2.7 Q2 Medicine

Journal of the American Society of Cytopathology Pub Date : 2026-09-01 Epub Date: 2026-04-25 DOI:10.1016/j.jasc.2026.04.005

Lakshmi Harinath MD, MPH, Esther Elishaev MD, Jonee Matsko SCT, MB, Amy Colaizzi SCT, Haley Brown CT, Rohit Bhargava MD, Liron Pantanowitz MD, PhD, MHA, Chengquan Zhao MD

{"title":"Determination of an optimal quality control method for Pap test analysis using digital cytology and artificial intelligence","authors":"Lakshmi Harinath MD, MPH, Esther Elishaev MD, Jonee Matsko SCT, MB, Amy Colaizzi SCT, Haley Brown CT, Rohit Bhargava MD, Liron Pantanowitz MD, PhD, MHA, Chengquan Zhao MD","doi":"10.1016/j.jasc.2026.04.005","DOIUrl":null,"url":null,"abstract":"<div><ul><li><span>•</span><span><div>The aim of this study was to determine an optimal method for performing quality control (QC) on digitized Pap tests when utilizing an artificial intelligence-assisted digital diagnostics system.</div></span></li><li><span>•</span><span><div>This study demonstrated strong interobserver agreement between pathologists and moderate consistency between individual QC reads carried out serially.</div></span></li><li><span>•</span><span><div>Digital QC checks can consist of randomly selecting a subset of cases, rescanning and re-analyzing with AI and acquiring interpretations from 2 or more CPs once regulatory guidelines are updated.</div></span></li></ul></div><div><h3>Introduction</h3><div>The aim of this study was to determine an optimal method for performing quality control on Pap tests that adhere to CLIA ‘88 mandates that are applicable to employing an artificial intelligence (AI)-based system in routine practice.</div></div><div>The aim of this study was to determine an optimal method for performing quality control on Pap tests that adhere to CLIA ‘88 mandates that are applicable to employing an artificial intelligence (AI)-based system in routine practice.</div><div><h3>Materials and methods</h3><div>Four hundred ninety-seven archival ThinPrep Pap slides were retrieved and scanned with Genius Digital Diagnostics System. A cytologist and 3 cytopathologists (CP) performed initial interpretations (first read). Subsequently, all slides were rescanned and the new output was re-reviewed by another cytologist and the same 3 CP’s after a washout period of 14 months (second read). The result of serial reads of each pathologist was compared with the original ThinPrep Interpretation (OTPI) and with each other. Interobserver concordance was calculated for all reads.</div></div><div>Four hundred ninety-seven archival ThinPrep Pap slides were retrieved and scanned with Genius Digital Diagnostics System. A cytologist and 3 cytopathologists (CP) performed initial interpretations (first read). Subsequently, all slides were rescanned and the new output was re-reviewed by another cytologist and the same 3 CP’s after a washout period of 14 months (second read). The result of serial reads of each pathologist was compared with the original ThinPrep Interpretation (OTPI) and with each other. Interobserver concordance was calculated for all reads.</div><div><h3>Results</h3><div>Out of 497 cases 32.6%, 17.9%, 17.9%, 6.8%, 24.1%, and 0.6% were interpreted as negative for intraepithelial lesion or malignancy, atypical squamous cells of undetermined significance, low-grade squamous intraepithelial, atypical squamous cells, cannot exclude a high-grade squamous intraepithelial lesion, high-grade squamous intraepithelial lesion/squamous cell carcinoma, and atypical glandular cell/adenocarcinoma in situ/adenocarcinoma per OTPI. Kendall’s coefficient for concordance amongst the 3 CP’s for serial reads were 0.924 and 0.892, respectively. Concordance between 4 diagnostic results (3 CP results and OTPI) for first and second reads was 0.902 and 0.865, respectively. Three hundred seven (61.8%), 310 (62.4%), and 328 (66%) cases were consistent when serial reads were compared for pathologist A, B, and C, respectively. Kappa values between reads ranged from 0.51 to 0.559 amongst the pathologists.</div></div><div>Out of 497 cases 32.6%, 17.9%, 17.9%, 6.8%, 24.1%, and 0.6% were interpreted as negative for intraepithelial lesion or malignancy, atypical squamous cells of undetermined significance, low-grade squamous intraepithelial, atypical squamous cells, cannot exclude a high-grade squamous intraepithelial lesion, high-grade squamous intraepithelial lesion/squamous cell carcinoma, and atypical glandular cell/adenocarcinoma in situ/adenocarcinoma per OTPI. Kendall’s coefficient for concordance amongst the 3 CP’s for serial reads were 0.924 and 0.892, respectively. Concordance between 4 diagnostic results (3 CP results and OTPI) for first and second reads was 0.902 and 0.865, respectively. Three hundred seven (61.8%), 310 (62.4%), and 328 (66%) cases were consistent when serial reads were compared for pathologist A, B, and C, respectively. Kappa values between reads ranged from 0.51 to 0.559 amongst the pathologists.</div><div><h3>Conclusions</h3><div>Regular quality control checks that involve rescanning a subset of randomly selected slides, reanalyzing them with AI and multiple interpretations from CP’s can be adopted for quality assurance for AI-assisted Pap test screening in the future after regulatory guidelines are updated.</div></div><div>Regular quality control checks that involve rescanning a subset of randomly selected slides, reanalyzing them with AI and multiple interpretations from CP’s can be adopted for quality assurance for AI-assisted Pap test screening in the future after regulatory guidelines are updated.</div>","PeriodicalId":38262,"journal":{"name":"Journal of the American Society of Cytopathology","volume":"15 5","pages":"Pages 369-377"},"PeriodicalIF":2.7000,"publicationDate":"2026-09-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the American Society of Cytopathology","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2213294526000608","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/4/25 0:00:00","PubModel":"Epub","JCR":"Q2","JCRName":"Medicine","Score":null,"Total":0}

引用次数: 0

Abstract

•
The aim of this study was to determine an optimal method for performing quality control (QC) on digitized Pap tests when utilizing an artificial intelligence-assisted digital diagnostics system.
•
This study demonstrated strong interobserver agreement between pathologists and moderate consistency between individual QC reads carried out serially.
•
Digital QC checks can consist of randomly selecting a subset of cases, rescanning and re-analyzing with AI and acquiring interpretations from 2 or more CPs once regulatory guidelines are updated.

Introduction

The aim of this study was to determine an optimal method for performing quality control on Pap tests that adhere to CLIA ‘88 mandates that are applicable to employing an artificial intelligence (AI)-based system in routine practice.

Materials and methods

Four hundred ninety-seven archival ThinPrep Pap slides were retrieved and scanned with Genius Digital Diagnostics System. A cytologist and 3 cytopathologists (CP) performed initial interpretations (first read). Subsequently, all slides were rescanned and the new output was re-reviewed by another cytologist and the same 3 CP’s after a washout period of 14 months (second read). The result of serial reads of each pathologist was compared with the original ThinPrep Interpretation (OTPI) and with each other. Interobserver concordance was calculated for all reads.

Results

Out of 497 cases 32.6%, 17.9%, 17.9%, 6.8%, 24.1%, and 0.6% were interpreted as negative for intraepithelial lesion or malignancy, atypical squamous cells of undetermined significance, low-grade squamous intraepithelial, atypical squamous cells, cannot exclude a high-grade squamous intraepithelial lesion, high-grade squamous intraepithelial lesion/squamous cell carcinoma, and atypical glandular cell/adenocarcinoma in situ/adenocarcinoma per OTPI. Kendall’s coefficient for concordance amongst the 3 CP’s for serial reads were 0.924 and 0.892, respectively. Concordance between 4 diagnostic results (3 CP results and OTPI) for first and second reads was 0.902 and 0.865, respectively. Three hundred seven (61.8%), 310 (62.4%), and 328 (66%) cases were consistent when serial reads were compared for pathologist A, B, and C, respectively. Kappa values between reads ranged from 0.51 to 0.559 amongst the pathologists.

Conclusions

Regular quality control checks that involve rescanning a subset of randomly selected slides, reanalyzing them with AI and multiple interpretations from CP’s can be adopted for quality assurance for AI-assisted Pap test screening in the future after regulatory guidelines are updated.

查看原文本刊更多论文

利用数字细胞学和人工智能确定巴氏试验分析的最佳质量控制方法。

简介：本研究的目的是确定一种执行巴氏试验质量控制的最佳方法，该方法符合CLIA’88的要求，适用于在常规实践中使用基于人工智能（AI）的系统。材料和方法：检索497份存档的ThinPrep Pap切片，用Genius数字诊断系统进行扫描。一名细胞学家和三名细胞病理学家（CP）进行了初步解释（一读）。随后，所有的切片重新扫描，新的输出由另一位细胞学家重新检查，冲洗期为14个月（第二次读取）后，同样的3次CP检查。将每位病理学家的连续读取结果与原始的ThinPrep Interpretation （OTPI）进行比较，并相互比较。计算所有读取的观察者间一致性。结果：在497例病例中，32.6%、17.9%、17.9%、6.8%、24.1%和0.6%的诊断为上皮内病变或恶性、意义不明的非典型鳞状细胞、低级别鳞状上皮内、非典型鳞状细胞、不能排除高级别鳞状上皮内病变、高级别鳞状上皮内病变/鳞状细胞癌、非典型腺细胞/原位腺癌/腺癌。序列序列的3个CP序列的肯德尔一致性系数分别为0.924和0.892。4项诊断结果（3项CP结果与OTPI）一、二reads的一致性分别为0.902和0.865。当对病理学家A、B和C的序列读数进行比较时，分别有307例（61.8%）、310例（62.4%）和328例（66%）的结果是一致的。病理组的Kappa值在0.51 ~ 0.559之间。结论：在监管指南更新后，可以采用定期的质量控制检查，包括重新扫描随机选择的切片子集，用人工智能和CP的多种解释重新分析它们，以保证未来人工智能辅助巴氏试验筛查的质量。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊