Testing the real-world utility of Bayes theorem in artificial intelligence-enabled electrocardiogram algorithm for the detection of left ventricular systolic dysfunction

Abstract

Objective

To assess how the theoretical principles of Bayes' theorem hold true in a clinically impactful way when testing the diagnostic performance of an artificial intelligence (AI) tool, using the case of the AI-enabled electrocardiogram (AI-ECG) screening tool that detects left ventricular systolic dysfunction (LVSD) in a “real-world” setting.

Patient and methods

We analyzed data from 42,883 consecutive patients who underwent a clinically indicated ECG and an echocardiogram within two weeks at our center between January 1st and December 31st^, 2019. We then evaluated area under the curve (AUC) of the receiver operating characteristics, sensitivity, specificity, positive and negative predictive values (PPV and NPV) of the AI-ECG to detect LVSD (left ventricle ejection fraction of ≤40 %) across (i) cumulative risk factor prevalence (pre-test probabilities) (ii) different diagnostic thresholds, using paired ECG-echocardiogram data.

Results

Prevalence of LVSD was 1.9 %, 4.0 %, 7.0 % and 13.9 % for patients with 0, 1–2, 3–4 and ≥5 risk-factors for LVSD. The AUC of the AI-ECG for each group was 0.955, 0.933, 0.901 and 0.886, respectively (p for trend<0.001). Pre-test probabilities hardly influenced sensitivity but did impact specificity. PPV was affected more than NPV, which was modestly altered. Thresholds impacted diagnostic performance parameters, although their effect on NPV at low pre-test probability was negligible.

Conclusion

In real world, pre-test probabilities/cumulative risk-factors of disease do affect specificity. Using different diagnostic thresholds yields the highest impact on algorithm performance. A Bayesian approach may enhance individualized diagnostic performance when implementing AI algorithms.