A comparison of techniques for predicting telehealth visit failure

Abstract

Objective

Telehealth is an increasingly important method for delivering care. Health systems lack the ability to accurately predict which telehealth visits will fail due to poor connection, poor technical literacy, or other reasons. This results in wasted resources and disrupted patient care. The purpose of this study is to characterize and compare various methods for predicting telehealth visit failure, and to determine the prediction method most suited for implementation in a real-time operational setting.

Methods

A single-center, retrospective cohort study was conducted using data sourced from our data warehouse. Patient demographic information and data characterizing prior visit success and engagement with electronic health tools were included. Three main model types were evaluated: an existing scoring model developed by Hughes et al., a regression-based scoring model, and Machine Learning classifiers. Variables were selected for their importance and anticipated availability; Number Needed to Treat was used to demonstrate the number of interventions (e.g. pre-visit phone calls) required to improve success rates in the context of weekly patient volumes.

Results

217, 229 visits spanning 480 days were evaluated, of which 22,443 (10.33 %) met criteria for failure. Hughes et al.’s model applied to our data yielded an Area Under the Receiver Operating Characteristics Curve (AUC ROC) of 0.678 when predicting failure. A score-based model achieved an AUC ROC of 0.698. Logistic Regression, Random Forest, and Gradient Boosting models demonstrated AUC ROCs ranging from 0.7877 to 0.7969. A NNT of 32 was achieved if the 263 highest-risk patients were selected in a low-volume week using the RF classifier, compared to an expected NNT of 90 if the same number of patients were randomly selected.

Conclusions

Machine Learning classifiers demonstrated superiority over score-based methods for predicting telehealth visit failure. Prospective evaluation is required; evaluation using NNT as a metric can help to operationalize these models.