Comparison of Machine Learning Models to a Novel Score in the Identification of Patients at Low Risk for Diabetic Retinopathy

Purpose

To develop an easily applicable predictor of patients at low risk for diabetic retinopathy (DR).

Design

An experimental study on the development and validation of machine learning models (MLMs) and a novel retinopathy risk score (RRS) to detect patients at low risk for DR.

Subjects

All individuals aged ≥18 years of age who participated in the telemedicine retinal screening initiative through Temple University Health Systems from October 1, 2016 through December 31, 2020. The subjects must have documented evidence of their diabetes mellitus (DM) diagnosis as well as a documented glycosylated hemoglobin (HbA1c) recorded in their chart within 6 months of the retinal screening photograph.

Methods

The charts of 1930 subjects (1590 evaluable) undergoing telemedicine screening for DR were reviewed, and 30 demographic and clinical parameters were collected. Diabetic retinopathy is a dichotomous variable where low risk is defined as no or mild retinopathy using the International Clinical Diabetic Retinopathy severity score. Five MLMs were trained to predict patients at low risk for DR using 1050 subjects and further underwent 10-fold cross validation to maximize its performance indicated by the area under the receiver operator characteristic curve (AUC). Additionally, a novel RRS is defined as the product of HbA1c closest to screening and years with DM. Retinopathy risk score was also applied to generate a predictive model.

Main Outcome Measures

The performance of the trained MLMs and the RRS model was compared using DeLong’s test. The models were further validated using a separate unseen test set of 540 subjects. The performance of the validation models were compared using DeLong’s test and chi-square tests.

Results

Using the test set, the AUC for the RRS was not statistically different from 4 out of 5 MLM. The error rate for predicting low-risk patients using the RRS was significantly lower than the naive rate (0.097 vs. 0.19; P < 0.0001), and it was comparable to the error rates of the MLMs.

Conclusions

This novel RRS is a potentially useful and easily deployable predictor of patients at low risk for DR.

Financial Disclosure(s)

Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.