Automated Technical Skill and Performance Assessment in Otology and Neurotology: A Scoping Review.

Abstract

Objectives/hypothesis: This scoping review aims to provide an overview of existing semi-automated and fully automated methods for technical skill and performance assessment in otologic and neurotologic procedures.

Study design: Scoping review.

Databases reviewed: Ovid MEDLINE (PubMed), Ovid EMBASE, Web of Science Core Collection, and IEEE Xplor Digital Library.

Methods: A literature search was conducted according to PRISMA-ScR. Included studies were full-text articles that detailed an automated method of technical skill and performance assessment in otologic/neurotologic procedures. Extracted elements included general study characteristics (publication year, study objective, validity type, surgical procedure, and setting) and assessment approach characteristics (method of analysis, metrics assessed, source of metric data, degree of automation, and use of artificial intelligence [AI]).

Results: A total of 1,141 studies were identified from the literature search. After deduplication, title/abstract screening, and full-text review, 21 studies met the inclusion criteria. All but one of the included studies focused on mastoidectomy. Most studies assessed performance exclusively in VR-simulated mastoidectomy (n = 12) as opposed to cadaveric, 3D-printed, or live dissections. The majority of studies concentrated on establishing internal validity of their assessment methods (n = 13). Performance metrics were primarily obtained through motion analysis and final product analysis. Only a minority of studies used AI, which typically involved machine learning regression or classification to predict skill levels based on automatically extracted metrics.

Conclusion: This scoping review explores the developing landscape of automated technical skill and performance assessment in otology and neurotology. Though progress has been made in automating assessment in the field, most investigations are narrowly focused on performance in VR-simulated mastoidectomy and lack external validity evidence. AI and computer vision (CV), which have advanced automated assessment in other surgical fields, have been underutilized in assessing performance in otology and neurotology. Future work must explore the development and validation of automated assessment approaches across a wider range of otologic and neurotologic procedures. Incorporation of novel AI/CV techniques may facilitate real-time integration of automated assessment in a broader range of simulated procedures and live surgical settings.