Karim A. Zaazoue MD , Mathew R. McCann MD , Ahmed K. Ahmed MD, MSc , Isabel O. Cortopassi MD , Young M. Erben MD , Brent P. Little MD , Justin T. Stowell MD , Beau B. Toskich MD , Charles A. Ritchie MD
{"title":"评估2019冠状病毒病患者对比增强计算机断层扫描和计算机断层肺血管造影自动检测肺栓塞的市售人工智能算法的性能","authors":"Karim A. Zaazoue MD , Mathew R. McCann MD , Ahmed K. Ahmed MD, MSc , Isabel O. Cortopassi MD , Young M. Erben MD , Brent P. Little MD , Justin T. Stowell MD , Beau B. Toskich MD , Charles A. Ritchie MD","doi":"10.1016/j.mayocpiqo.2023.03.001","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><p>To investigate the performance of a commercially available artificial intelligence (AI) algorithm for the detection of pulmonary embolism (PE) on contrast-enhanced computed tomography (CT) scans in patients hospitalized for coronavirus disease 2019 (COVID-19).</p></div><div><h3>Patients and Methods</h3><p>Retrospective analysis was performed of all contrast-enhanced chest CT scans of patients admitted for COVID-19 between March 1, 2020 and December 31, 2021. Based on the original radiology reports, all PE-positive examinations were included (n=527). Using a reversed-flow single-gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative examinations (n=977) was included. Pulmonary parenchymal disease severity was assessed for all the included studies using a semiquantitative system, the total severity score. All included CT scans were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by 3 blinded radiologists, who rendered a final determination of indeterminate, positive, or negative.</p></div><div><h3>Results</h3><p>A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and were excluded. The sensitivity and specificity of AI were 93.2% (95% CI, 90.6%-95.2%) and 99.6% (95% CI, 98.9%-99.9%), respectively. The accuracy of AI for all total severity score groups (mild, moderate, and severe) was high (98.4%, 96.7%, and 97.2%, respectively). Artificial intelligence was more accurate in PE detection on CT pulmonary angiography scans than on contrast-enhanced CT scans (<em>P</em><.001), with an optimal Hounsfield unit of 362 (<em>P</em>=.048).</p></div><div><h3>Conclusion</h3><p>The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast-enhanced CT scans in patients with COVID-19 regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.</p></div>","PeriodicalId":94132,"journal":{"name":"Mayo Clinic proceedings. Innovations, quality & outcomes","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2023-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9995315/pdf/","citationCount":"1","resultStr":"{\"title\":\"Evaluating the Performance of a Commercially Available Artificial Intelligence Algorithm for Automated Detection of Pulmonary Embolism on Contrast-Enhanced Computed Tomography and Computed Tomography Pulmonary Angiography in Patients With Coronavirus Disease 2019\",\"authors\":\"Karim A. Zaazoue MD , Mathew R. McCann MD , Ahmed K. Ahmed MD, MSc , Isabel O. Cortopassi MD , Young M. Erben MD , Brent P. Little MD , Justin T. Stowell MD , Beau B. Toskich MD , Charles A. Ritchie MD\",\"doi\":\"10.1016/j.mayocpiqo.2023.03.001\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<div><h3>Objective</h3><p>To investigate the performance of a commercially available artificial intelligence (AI) algorithm for the detection of pulmonary embolism (PE) on contrast-enhanced computed tomography (CT) scans in patients hospitalized for coronavirus disease 2019 (COVID-19).</p></div><div><h3>Patients and Methods</h3><p>Retrospective analysis was performed of all contrast-enhanced chest CT scans of patients admitted for COVID-19 between March 1, 2020 and December 31, 2021. Based on the original radiology reports, all PE-positive examinations were included (n=527). Using a reversed-flow single-gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative examinations (n=977) was included. Pulmonary parenchymal disease severity was assessed for all the included studies using a semiquantitative system, the total severity score. All included CT scans were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by 3 blinded radiologists, who rendered a final determination of indeterminate, positive, or negative.</p></div><div><h3>Results</h3><p>A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and were excluded. The sensitivity and specificity of AI were 93.2% (95% CI, 90.6%-95.2%) and 99.6% (95% CI, 98.9%-99.9%), respectively. The accuracy of AI for all total severity score groups (mild, moderate, and severe) was high (98.4%, 96.7%, and 97.2%, respectively). Artificial intelligence was more accurate in PE detection on CT pulmonary angiography scans than on contrast-enhanced CT scans (<em>P</em><.001), with an optimal Hounsfield unit of 362 (<em>P</em>=.048).</p></div><div><h3>Conclusion</h3><p>The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast-enhanced CT scans in patients with COVID-19 regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.</p></div>\",\"PeriodicalId\":94132,\"journal\":{\"name\":\"Mayo Clinic proceedings. Innovations, quality & outcomes\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2023-06-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9995315/pdf/\",\"citationCount\":\"1\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Mayo Clinic proceedings. Innovations, quality & outcomes\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://www.sciencedirect.com/science/article/pii/S2542454823000115\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Mayo Clinic proceedings. Innovations, quality & outcomes","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2542454823000115","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
Evaluating the Performance of a Commercially Available Artificial Intelligence Algorithm for Automated Detection of Pulmonary Embolism on Contrast-Enhanced Computed Tomography and Computed Tomography Pulmonary Angiography in Patients With Coronavirus Disease 2019
Objective
To investigate the performance of a commercially available artificial intelligence (AI) algorithm for the detection of pulmonary embolism (PE) on contrast-enhanced computed tomography (CT) scans in patients hospitalized for coronavirus disease 2019 (COVID-19).
Patients and Methods
Retrospective analysis was performed of all contrast-enhanced chest CT scans of patients admitted for COVID-19 between March 1, 2020 and December 31, 2021. Based on the original radiology reports, all PE-positive examinations were included (n=527). Using a reversed-flow single-gate diagnostic accuracy case-control model, a randomly selected cohort of PE-negative examinations (n=977) was included. Pulmonary parenchymal disease severity was assessed for all the included studies using a semiquantitative system, the total severity score. All included CT scans were sent for interpretation by the commercially available AI algorithm, Aidoc. Discrepancies between AI and original radiology reports were resolved by 3 blinded radiologists, who rendered a final determination of indeterminate, positive, or negative.
Results
A total of 78 studies were found to be discrepant, of which 13 (16.6%) were deemed indeterminate by readers and were excluded. The sensitivity and specificity of AI were 93.2% (95% CI, 90.6%-95.2%) and 99.6% (95% CI, 98.9%-99.9%), respectively. The accuracy of AI for all total severity score groups (mild, moderate, and severe) was high (98.4%, 96.7%, and 97.2%, respectively). Artificial intelligence was more accurate in PE detection on CT pulmonary angiography scans than on contrast-enhanced CT scans (P<.001), with an optimal Hounsfield unit of 362 (P=.048).
Conclusion
The AI algorithm demonstrated high sensitivity, specificity, and accuracy for PE on contrast-enhanced CT scans in patients with COVID-19 regardless of parenchymal disease. Accuracy was significantly affected by the mean attenuation of the pulmonary vasculature. How this affects the legitimacy of the binary outcomes reported by AI is not yet known.