Natural Language Processing and soft data for motor skill assessment: A case study in surgical training simulations

Abstract

Background and Objective:

Automated surgical skill assessment using kinematic and video data (hard data) sources has been widely adopted in the literature. However, experts’ opinions (soft data) in the form of free-text could be an invaluable source for evaluating one’s skill level since the availability and semantic richness of the soft data are both higher than the hard data. In this paper, the feasibility of using soft data as a single source of skill assessment is analyzed with various Natural Language Processing (NLP) algorithms of different levels of complexity.

Methods:

An experiment named “Vertex Pursuit” was designed to address the absence of a dataset with free-text soft data in synchronization with hard data. This experiment challenges participants’ hand-eye coordination, both-hand coordination, precision, and dexterity by tracking haptic device movements along a star pentagon shape. Top-performing participants receive additional training to provide expert feedback through free-text comments evaluating their peers’ trials. Traditional machine learning approaches are employed, including various word and sentence embedding techniques combined with a diverse set of classifiers, to assess skill levels based on this soft data. Additionally, encoder-only and decoder-only large language models (LLMs) are applied to the data, with the latter leveraging three prompt engineering techniques.

Results:

The task of skill assessment using soft data is demonstrated to be a complex NLP task, and as the complexity of the method increases, the results improve. The top performance was achieved with the decoder-only LLMs and the rule-based prompting strategy.

Conclusion:

This paper studied the feasibility of using soft data in a simulated surgical skill assessment scenario. While further research is needed, the proposed methods can reduce subjectivity, alleviate the burden on human experts, and enable more widespread, scalable skill evaluation in surgical training programs.