Effect of Artificial Intelligence-Augmented Human Instruction on Feedback Frequency and Surgical Performance During Simulation Training.

Abstract

Objective: To determine whether personalized feedback from a human instructor receiving artificial intelligence (AI) error data will result in reduced feedback frequency and improvement of surgical skill compared to AI instruction. As feedback was only provided following AI error detection, a reduced feedback frequency is associated with fewer errors in performance. We hypothesized that AI-augmented personalized instruction would result in reduced feedback frequency and improvement in technical skill.

Design: This cross-sectional cohort study was a follow-up of a randomized controlled trial, which utilized the NeuroVR, an immersive virtual reality neurosurgical simulator. Participants were stratified by year in medical school and block randomized to receive one of 3 educational interventions as they performed simulated procedures on the NeuroVR: AI tutor instruction, scripted human instruction, and AI-augmented personalized instruction. Performance was assessed by the feedback frequency and technical skill performance metrics.

Clinicaltrials: gov ID: NCT06273579.

Setting: Neurosurgical Simulation and Artificial Intelligence Learning Centre, McGill University, Montreal, Canada.

Participants: Volunteer sample of medical students from 4 Quebec universities in preparatory, first, or second year without prior use of the NeuroVR. Eighty-eight students participated in the study with 87 included in the final analysis; 1 was excluded due to technical issues.

Results: By the third repetition, the AI-augmented personalized instruction group received significantly fewer total instructions (incidence rate ratio [IRR], 1.50 [95% CI, 1.16-1.94] instructions; p < 0.001), and high aspirator force instructions (IRR, 1.71 [95% CI, 1.15-2.55] instructions; p = 0.002), compared to the second repetition. Compared to AI tutor instruction, AI-augmented personalized instruction resulted in a significantly lower rate of healthy tissue removal (p = 0.01), instrument tip separation distance (mean ratio, 1.25 [95% CI, 1.05-1.50] mm; p = 0.008), and aspirator force (mean ratio, 1.68 [95% CI, 1.23-2.31] N; p < 0.001). AI-augmented personalized instruction showed a significant improvement from baseline in all subsequent repetitions for all performance metrics.

Conclusions: This cohort study demonstrated that AI-augmented personalized instruction resulted in less frequent feedback, indicating fewer errors in trainee performance, and an improvement in simulated surgical skills.