A Statistical Framework to Detect and Quantify Operator-Learning Curves in Medical Device Safety Evaluation.

Importance: Safety issues leading to patient harm and significant costs have been identified in several post-market medical devices. Recently, powerful learning effects (LE) have been documented in numerous medical devices. Correctly attributing safety signals to learning or device effects allows for appropriate corrective actions and recommendations to improve patient safety.

Objective: To develop and assess the statistical performance of an analytic framework to detect the presence of LE and quantify the learning curve (LC).

Design and setting: We generated synthetic datasets based on observed clinical distributions and complex feature correlations among patients hospitalized at US Department of Veterans Affairs facilities. Each dataset represents a hypothetical early experience in the use of high-risk medical devices, with a device of interest and a reference device. The study blinded the analysis team to the data-generation process.

Methods: We developed predictive models using generalized additive models and estimated LC parameters using the Levenberg-Marqualdt algorithm. We evaluated the performance using sensitivity, specificity, and likelihood ratio (LR) in detecting the presence of LE and, if present, the goodness-of-fit of the estimated LC based on the root-mean squared error.

Results: Among the 2483 simulated datasets, the median (IQR) number of cases was 218,000 (116,000-353,000). LE were detected in 2065 of the 2291 datasets for which learning was specified (sensitivity: 90%; specificity: 88%; LR: 7). We adequately estimated the LC in 1632 (81%) of the 2013 datasets in which LE was detected and estimated LC.

Discussion: This study demonstrated the framework to be robust in disentangling LE from device safety signals and in estimating LC.

Conclusion: In medical device safety evaluation, the operator-learning effects associated with the safety of medical devices can be effectively modeled and characterized. This study warrants subsequent framework validation by using real-world clinical datasets.