Clinical Performance Evaluation of an Artificial Intelligence-Based Tool for Predicting the Presence of Obstructive Coronary Artery Disease: Protocol for a Cohort Observational Study.

Abstract

Background: A significant number of individuals undergoing coronary computed tomography angiography (CCTA) for suspected (CAD) have nonobstructive or no CAD. There is a need for clinically proven models that can predict the pretest probability of stable CAD and help to identify low-risk individuals. Optimizing patient stratification is of paramount importance to improve diagnostic yield and cost-effectiveness.

Objective: We aimed to determine whether each patient needs to undergo CCTA because of suspected CAD. The main objective of this study is to evaluate the clinical performance of an artificial intelligence (AI)-based tool in predicting significant coronary artery stenosis (>50%), as well as its utility by medical professionals.

Methods: Data for this study have been acquired from 750 participants as part of routine clinical practice in AHEPA (American Hellenic Educational Progressive Association) General Hospital of Thessaloniki. The dataset has several features, including demographics (eg, age, gender), medical history (eg, diabetes mellitus, arterial hypertension), and clinical variables (eg, creatinine, epicardial fat volume). At least 2 expert cardiologists and 2 expert radiologists are involved in this study, who provide the ground truth. A trained AI-based model embedded in an easy-to-use and user-friendly web application is implemented in practice. Several AI algorithms are being examined, and the model found to perform best so far is the Optimized Voting model, which is a combination of the best performing iterations of random forest and extreme gradient boosting. The performance metrics that are being used are accuracy, precision, recall, F₁-score, area under the receiver operating characteristic curve, and area under the precision-recall curve.

Results: Recruitment for this study began in July 2023. Data collection, development, training, and deployment of the AI web tool were completed by May 2024. In total, data from 500 individuals were collected for training and internal validation, while the best performing model was validated externally in another 250 individuals. For training and internal validation, the dataset was split into 70% for training and 20% for validation and 10% for testing. Currently, the best performing model achieves an accuracy of approximately 82% in successfully predicting stenosis greater than 50%. Additionally, an explainable AI algorithm is used to provide explanations in relation to the decisions made aiming to increase the trust of the clinicians in the tool.

Conclusions: The proposed study represents a novel approach of a web-based AI-driven solution with explainability features for optimizing patient stratification with the goal of improving diagnostic yield and cost-effectiveness of CCTA utilization within the context of cardiology clinical practice.

International registered report identifier (irrid): DERR1-10.2196/67697.