The use of generative artificial intelligence-based dictation in a neurosurgical practice: a pilot study.

Abstract

Objective: Document dictation remains a significant clinical burden and generative artificial intelligence (AI) systems utilizing transformer-based technology offer efficient speech processing methods that could streamline clinical documentation. This study aimed to evaluate the potential of generative AI in enhancing dictation efficiency and workflow within a targeted neurosurgical practice.

Methods: Ten operative reports from both cranial and spinal neurosurgical procedures were dictated and recorded by three independent physicians. The audio files were processed by 1) a modified speech-to-text model implemented based on a backbone architecture created by OpenAI's Whisper model and 2) Nuance's Dragon Medical One as a comparative commercial standard. Word error rate (WER) was manually reviewed.

Results: The mean WER was 1.75% for Whisper and 1.54% for Dragon (p = 0.080). When excluding linguistic errors, Whisper outperformed Dragon with a mean WER of 0.50% versus 1.34% (p < 0.001), including the mean number of total errors (Whisper: 6.1, Dragon: 9.7; p = 0.002). For all unstratified dictations, a positive correlation was seen between total errors and word count (p < 0.001, R2 = 0.37), as well as total errors and recording length (p < 0.001, R2 = 0.22). A positive correlation was noted between words spoken per second and total errors for Dragon (p = 0.020, R2 = 0.18), but not for Whisper (p = 0.205, R2 = 0.06). Similarly, when analyzing linguistic errors only, this trend held for Dragon (p = 0.014, R2 = 0.20), but not for Whisper (p = 0.331, R2 = 0.03).

Conclusions: An AI-based model performed at a noninferior rate compared to a commercially available speech-to-text dictation program. Generative models provide potential benefits such as contextual inference that show promise in limiting errors with increased dictation speed or adjustment for impure input data.