Data-Driven Metrics of Operative Performance: The Technology-Enhanced Surgical Training (TEST) Delphi Consensus Study

IF 2.1 3区医学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

Journal of Surgical Education Pub Date : 2025-09-18 DOI:10.1016/j.jsurg.2025.103701

Lachlan Dick MBChB , Victoria Ruth Tallentire MD , Annemarie B. Docherty PhD , Douglas S. Smink MD, MPH , Richard JE Skipworth MD , Steven Yule PhD , Technology-Enhanced Surgical Training (TEST) Consensus Group

{"title":"Data-Driven Metrics of Operative Performance: The Technology-Enhanced Surgical Training (TEST) Delphi Consensus Study","authors":"Lachlan Dick MBChB , Victoria Ruth Tallentire MD , Annemarie B. Docherty PhD , Douglas S. Smink MD, MPH , Richard JE Skipworth MD , Steven Yule PhD , Technology-Enhanced Surgical Training (TEST) Consensus Group","doi":"10.1016/j.jsurg.2025.103701","DOIUrl":null,"url":null,"abstract":"<div><h3>Objectives</h3><div>Technology-enhanced assessment of operative performance offers greater objectivity and reliability than traditional methods. However, consensus on the most critical metrics for informing the future of surgical training is lacking, and guidance on optimal application of these analytics to enhance feedback is needed. This study aimed to establish consensus on advanced operative performance metrics in surgical training and identify applications to enhance feedback.</div></div><div><h3>Design</h3><div>Data-driven 3-round Delphi method, where experts iteratively rated performance metrics derived from the surgical literature on a 7-point Likert scale. Statements reaching the <em>a priori</em> consensus threshold [round 1: > 75% rating => 6; round 2 + 3 > 50% rating => 6] were included in the final consensus statement.</div></div><div><h3>Setting</h3><div>Electronic survey, using the research electronic data capture (REDCap) system.</div></div><div><h3>Participants</h3><div>A pan-specialty, international panel of 57 surgical trainers, trainees and researchers.</div></div><div><h3>Results</h3><div>Twenty-two statements met the consensus threshold, of which 10 represented individual training specific metrics. Technical metrics were: dissection in the correct tissue plane (58.1% consensus); economy of motion (58.1%); and technical errors (53.5%). Non-technical metrics were: situation awareness (60.5%); communication (55.8%); decision -making (51.2%); and cognitive load (51.2%). Outcome metrics included: a safety score (60.5%); duration to react to adverse events (60.5%); and a global performance score (53.5%). Suggested applications to surgical training included: comparing individual metrics over time (65.1%); benchmarking against the average performance of trainees with similar experience (55.8%); and guiding formative assessments (51.2%).</div></div><div><h3>Conclusions</h3><div>This Delphi study established international consensus on advanced operative performance metrics, providing a foundation for improved feedback in surgical training. Demonstrating validity of existing metrics and developing novel ones that were highly ranked are the critical next steps to integrate advanced technologies into surgical training curricula, enhancing trainee development and patient safety.</div></div>","PeriodicalId":50033,"journal":{"name":"Journal of Surgical Education","volume":"82 11","pages":"Article 103701"},"PeriodicalIF":2.1000,"publicationDate":"2025-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Surgical Education","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S193172042500282X","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives

Technology-enhanced assessment of operative performance offers greater objectivity and reliability than traditional methods. However, consensus on the most critical metrics for informing the future of surgical training is lacking, and guidance on optimal application of these analytics to enhance feedback is needed. This study aimed to establish consensus on advanced operative performance metrics in surgical training and identify applications to enhance feedback.

Design

Data-driven 3-round Delphi method, where experts iteratively rated performance metrics derived from the surgical literature on a 7-point Likert scale. Statements reaching the a priori consensus threshold [round 1: > 75% rating => 6; round 2 + 3 > 50% rating => 6] were included in the final consensus statement.

Setting

Electronic survey, using the research electronic data capture (REDCap) system.

Participants

A pan-specialty, international panel of 57 surgical trainers, trainees and researchers.

Results

Twenty-two statements met the consensus threshold, of which 10 represented individual training specific metrics. Technical metrics were: dissection in the correct tissue plane (58.1% consensus); economy of motion (58.1%); and technical errors (53.5%). Non-technical metrics were: situation awareness (60.5%); communication (55.8%); decision -making (51.2%); and cognitive load (51.2%). Outcome metrics included: a safety score (60.5%); duration to react to adverse events (60.5%); and a global performance score (53.5%). Suggested applications to surgical training included: comparing individual metrics over time (65.1%); benchmarking against the average performance of trainees with similar experience (55.8%); and guiding formative assessments (51.2%).

Conclusions

This Delphi study established international consensus on advanced operative performance metrics, providing a foundation for improved feedback in surgical training. Demonstrating validity of existing metrics and developing novel ones that were highly ranked are the critical next steps to integrate advanced technologies into surgical training curricula, enhancing trainee development and patient safety.

查看原文本刊更多论文

手术表现的数据驱动指标：技术增强手术训练（TEST）德尔菲共识研究。

目的：技术增强的手术绩效评估比传统方法更具客观性和可靠性。然而，对于告知外科培训未来的最关键指标缺乏共识，需要对这些分析的最佳应用进行指导，以增强反馈。本研究旨在就外科培训中先进的手术表现指标建立共识，并确定增强反馈的应用。设计：数据驱动的3轮德尔菲法，专家们根据7分李克特量表对外科文献中的表现指标进行迭代评分。达到先验共识阈值的陈述[第1轮：> 75%评级=> 6；第2轮 + 3 > 50%评级=> 6]被纳入最终共识声明。设置：电子调查，使用研究电子数据捕获（REDCap）系统。参与者：由57名外科培训师、学员和研究人员组成的泛专业国际小组。结果：22个陈述符合共识阈值，其中10个代表个人训练特定指标。技术指标为：正确组织平面解剖（58.1%一致）；运动经济性（58.1%）；技术错误（53.5%）。非技术指标为：态势感知（60.5%）；交流(55.8%);决策（51.2%）；认知负荷（51.2%）。结局指标包括：安全评分（60.5%）；不良事件反应持续时间（60.5%）；以及全球表现得分（53.5%）。建议应用于外科培训包括：比较个体指标随时间的变化（65.1%）；以具有相似经验的学员的平均表现为基准（55.8%）；指导形成性评估（51.2%）。结论：本德尔菲研究建立了先进手术表现指标的国际共识，为改进手术培训反馈提供了基础。证明现有指标的有效性和开发排名靠前的新指标是将先进技术整合到外科培训课程、加强培训生发展和患者安全的关键步骤。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Surgical Education EDUCATION, SCIENTIFIC DISCIPLINES-SURGERY

CiteScore

5.60

自引率

10.30%

发文量

261

审稿时长

48 days

期刊介绍： The Journal of Surgical Education (JSE) is dedicated to advancing the field of surgical education through original research. The journal publishes research articles in all surgical disciplines on topics relative to the education of surgical students, residents, and fellows, as well as practicing surgeons. Our readers look to JSE for timely, innovative research findings from the international surgical education community. As the official journal of the Association of Program Directors in Surgery (APDS), JSE publishes the proceedings of the annual APDS meeting held during Surgery Education Week.