Variability in Testing Scenario Complexity Reduces the Validity of the Advanced Trauma Life Support Certification Assessment

IF 2.6 3区医学 Q1 EDUCATION, SCIENTIFIC DISCIPLINES

Journal of Surgical Education Pub Date : 2025-05-30 DOI:10.1016/j.jsurg.2025.103553

Seguin Ryerson MD, Peabody Jeremy MD, Ryan Joanna F MD, MEd, FRCSC, Anantha Ram MD, FRCSC, Turner Simon R MD, MEd, FRCSC, Strickland Matt MD, FRCSC

{"title":"Variability in Testing Scenario Complexity Reduces the Validity of the Advanced Trauma Life Support Certification Assessment","authors":"Seguin Ryerson MD, Peabody Jeremy MD, Ryan Joanna F MD, MEd, FRCSC, Anantha Ram MD, FRCSC, Turner Simon R MD, MEd, FRCSC, Strickland Matt MD, FRCSC","doi":"10.1016/j.jsurg.2025.103553","DOIUrl":null,"url":null,"abstract":"<div><h3>BACKGROUND</h3><div>The Advanced Trauma Life Support (ATLS) course plays a pivotal role in the trauma education of healthcare professionals on a global scale. To achieve certification, participants are evaluated in simulated trauma scenarios. The McGill Simulation Complexity Score (MSCS) is a trauma-specific tool with supporting validity evidence, developed to quantify the complexity of trauma simulations. This study aims to assess the heterogeneity and complexity of the trauma scenarios used to assess ATLS candidates.</div></div><div><h3>METHODS</h3><div>Four independent reviewers utilized the MSCS to assess the 21 ATLS assessment scenarios for complexity. Each resuscitation component (Airway, Breathing, Circulation, Disability, Exposure) was given a score from 0 (lowest complexity) to 4 (highest complexity), with a total case score between 0 and 20. Variability between case complexity and inter-reviewer reliability was assessed.</div></div><div><h3>RESULTS</h3><div>The mean complexity score was 7.7 (standard deviation 0.61), with scores ranging from 3.5 to 13.25. Inter-rater reliability was high with an interclass correlation coefficient of 0.80. Of the individual resuscitation elements scored, the circulation component demonstrated the highest average complexity while the exposure component contributed the least complexity with mean scores of 2.38 and 0.82, respectively.</div></div><div><h3>CONCLUSION</h3><div>There is variability in the complexity scores of the ATLS assessment simulation scenarios. This variability is a threat to the validity of the ATLS course assessment. Given the requirement for ATLS certification in many health professions training programs and employment opportunities, reform of course assessment scenarios should be considered to standardize complexity and support assessment validity.</div></div>","PeriodicalId":50033,"journal":{"name":"Journal of Surgical Education","volume":"82 8","pages":"Article 103553"},"PeriodicalIF":2.6000,"publicationDate":"2025-05-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Surgical Education","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1931720425001345","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

BACKGROUND

The Advanced Trauma Life Support (ATLS) course plays a pivotal role in the trauma education of healthcare professionals on a global scale. To achieve certification, participants are evaluated in simulated trauma scenarios. The McGill Simulation Complexity Score (MSCS) is a trauma-specific tool with supporting validity evidence, developed to quantify the complexity of trauma simulations. This study aims to assess the heterogeneity and complexity of the trauma scenarios used to assess ATLS candidates.

METHODS

Four independent reviewers utilized the MSCS to assess the 21 ATLS assessment scenarios for complexity. Each resuscitation component (Airway, Breathing, Circulation, Disability, Exposure) was given a score from 0 (lowest complexity) to 4 (highest complexity), with a total case score between 0 and 20. Variability between case complexity and inter-reviewer reliability was assessed.

RESULTS

The mean complexity score was 7.7 (standard deviation 0.61), with scores ranging from 3.5 to 13.25. Inter-rater reliability was high with an interclass correlation coefficient of 0.80. Of the individual resuscitation elements scored, the circulation component demonstrated the highest average complexity while the exposure component contributed the least complexity with mean scores of 2.38 and 0.82, respectively.

CONCLUSION

There is variability in the complexity scores of the ATLS assessment simulation scenarios. This variability is a threat to the validity of the ATLS course assessment. Given the requirement for ATLS certification in many health professions training programs and employment opportunities, reform of course assessment scenarios should be considered to standardize complexity and support assessment validity.

查看原文本刊更多论文

测试场景复杂性的可变性降低了高级创伤生命支持认证评估的有效性

高级创伤生命支持（ATLS）课程在全球范围内的医疗保健专业人员的创伤教育中起着关键作用。为了获得认证，参与者在模拟的创伤场景中进行评估。McGill模拟复杂性评分（MSCS）是一种创伤特异性工具，具有支持有效性证据，用于量化创伤模拟的复杂性。本研究旨在评估用于评估ATLS候选人的创伤情景的异质性和复杂性。方法4位独立审稿人利用MSCS对21种ATLS评估方案的复杂性进行评估。每个复苏组成部分（气道、呼吸、循环、残疾、暴露）评分从0（最低复杂程度）到4（最高复杂程度），总病例评分在0到20之间。评估了病例复杂性和审稿人间可靠性之间的可变性。结果患者复杂性评分平均为7.7分（标准差为0.61），评分范围为3.5 ~ 13.25分。信度较高，类间相关系数为0.80。在评分的各个复苏要素中，循环成分的平均复杂性最高，暴露成分的平均复杂性最低，分别为2.38和0.82。结论ATLS评估模拟情景的复杂性评分存在差异。这种可变性对ATLS课程评估的有效性构成威胁。鉴于许多卫生专业培训项目和就业机会对ATLS认证的要求，应考虑改革课程评估方案，以规范复杂性并支持评估的有效性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Surgical Education EDUCATION, SCIENTIFIC DISCIPLINES-SURGERY

CiteScore

5.60

自引率

10.30%

发文量

261

审稿时长

48 days

期刊介绍： The Journal of Surgical Education (JSE) is dedicated to advancing the field of surgical education through original research. The journal publishes research articles in all surgical disciplines on topics relative to the education of surgical students, residents, and fellows, as well as practicing surgeons. Our readers look to JSE for timely, innovative research findings from the international surgical education community. As the official journal of the Association of Program Directors in Surgery (APDS), JSE publishes the proceedings of the annual APDS meeting held during Surgery Education Week.