Reliability and validity of simulation-based Electrocardiogram assessment rubrics for cardiac life support skills among medical students using generalizability theory.

IF 3.8 2区 医学 Q1 EDUCATION & EDUCATIONAL RESEARCH
Medical Education Online Pub Date : 2025-12-01 Epub Date: 2025-03-23 DOI:10.1080/10872981.2025.2479962
Sethapong Lertsakulbunlue, Kaophiphat Thammasoon, Anupong Kantiwong
{"title":"Reliability and validity of simulation-based Electrocardiogram assessment rubrics for cardiac life support skills among medical students using generalizability theory.","authors":"Sethapong Lertsakulbunlue, Kaophiphat Thammasoon, Anupong Kantiwong","doi":"10.1080/10872981.2025.2479962","DOIUrl":null,"url":null,"abstract":"<p><p>Simulation-based learning (SBL) is effective for EKG interpretation training in the advanced cardiac life support (ACLS) context, enhancing motivation, confidence, and learning outcomes. However, research on the psychometrics of assessment rubrics for ACLS skills among pre-clinical students is limited. This study investigates the validity and reliability of assessment rubrics for ACLS skills, including EKG interpretation, scenario and pharmacological management, and teamwork. An SBL course that integrates basic EKG interpretation into ACLS Stations was conducted at Phramongkutklao College of Medicine, utilizing high-fidelity mannequins to simulate realistic scenarios, enrolling 96 medical students. The course consisted of five independent stations, and each student was assessed once by two raters using ten-item assessment rubrics. The rubrics included three domains: (1) EKG and ACLS algorithm skills, (2) management and mechanisms of action, and (3) affective domains. Validity evidence on the content was gathered, and construct validity was confirmed with confirmatory factor analysis (CFA). Inter-rater and internal consistency reliability were calculated. Generalizability theory was utilized to analyse the data. Three expert reviews yielded an item-objective congruence index of 0.67-1.00, with iterative validation through alpha and beta tests. The CFA demonstrated a good fit, but two questions with loading factors below 0.30 were removed, resulting in an eight-item assessment form. An inter-rater correlation of 0.70 (p < 0.001) and a Cronbach's alpha of 0.76 was demonstrated. To achieve a Phi-coefficient ≥0.80, three raters and at least 10 items are required in a p×i×r crossed design. With eight items, r:(p×i) nested design reliability was 0.69, 0.79, and 0.83 for one, two, and three raters, respectively. While a single rater with 10 items achieved a Phi-coefficient of 0.74. The rubrics for assessing ACLS skills among pre-clinical students demonstrated acceptable validity and reliability. A condensed eight-item rubric with acceptable reliability is proposed as a practical tool for optimizing assessment in future evaluations relevant to the pre-clinical context.</p>","PeriodicalId":47656,"journal":{"name":"Medical Education Online","volume":"30 1","pages":"2479962"},"PeriodicalIF":3.8000,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11934178/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Education Online","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1080/10872981.2025.2479962","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/3/23 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0

Abstract

Simulation-based learning (SBL) is effective for EKG interpretation training in the advanced cardiac life support (ACLS) context, enhancing motivation, confidence, and learning outcomes. However, research on the psychometrics of assessment rubrics for ACLS skills among pre-clinical students is limited. This study investigates the validity and reliability of assessment rubrics for ACLS skills, including EKG interpretation, scenario and pharmacological management, and teamwork. An SBL course that integrates basic EKG interpretation into ACLS Stations was conducted at Phramongkutklao College of Medicine, utilizing high-fidelity mannequins to simulate realistic scenarios, enrolling 96 medical students. The course consisted of five independent stations, and each student was assessed once by two raters using ten-item assessment rubrics. The rubrics included three domains: (1) EKG and ACLS algorithm skills, (2) management and mechanisms of action, and (3) affective domains. Validity evidence on the content was gathered, and construct validity was confirmed with confirmatory factor analysis (CFA). Inter-rater and internal consistency reliability were calculated. Generalizability theory was utilized to analyse the data. Three expert reviews yielded an item-objective congruence index of 0.67-1.00, with iterative validation through alpha and beta tests. The CFA demonstrated a good fit, but two questions with loading factors below 0.30 were removed, resulting in an eight-item assessment form. An inter-rater correlation of 0.70 (p < 0.001) and a Cronbach's alpha of 0.76 was demonstrated. To achieve a Phi-coefficient ≥0.80, three raters and at least 10 items are required in a p×i×r crossed design. With eight items, r:(p×i) nested design reliability was 0.69, 0.79, and 0.83 for one, two, and three raters, respectively. While a single rater with 10 items achieved a Phi-coefficient of 0.74. The rubrics for assessing ACLS skills among pre-clinical students demonstrated acceptable validity and reliability. A condensed eight-item rubric with acceptable reliability is proposed as a practical tool for optimizing assessment in future evaluations relevant to the pre-clinical context.

利用概括性理论对医科学生的心脏生命支持技能进行基于模拟心电图的评估标准的可靠性和有效性。
基于模拟的学习(SBL)对于高级心脏生命支持(ACLS)环境下的心电图解释训练是有效的,可以增强动机、信心和学习结果。然而,临床前学生ACLS技能测评指标的心理测量学研究有限。本研究探讨了ACLS技能评估标准的效度和信度,包括心电图解释、情景和药物管理以及团队合作。Phramongkutklao医学院开设了一门SBL课程,将基本的心电图解释整合到ACLS站中,利用高保真人体模型模拟现实场景,招收了96名医学生。课程由五个独立的站点组成,每个学生由两个评分员使用十项评估标准进行一次评估。这些标准包括三个领域:(1)心电图和ACLS算法技能;(2)管理和作用机制;(3)情感领域。收集内容的效度证据,并采用验证性因子分析(CFA)对结构进行效度验证。计算了内部和内部一致性信度。利用概化理论对数据进行分析。三次专家评审得出的项目-目标一致性指数为0.67-1.00,并通过alpha和beta测试进行了迭代验证。CFA表现出了良好的拟合性,但删除了加载因子低于0.30的两个问题,形成了一个包含8个项目的评估表格。比率间相关系数为0.70 (p
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Medical Education Online
Medical Education Online EDUCATION & EDUCATIONAL RESEARCH-
CiteScore
6.00
自引率
2.20%
发文量
97
审稿时长
8 weeks
期刊介绍: Medical Education Online is an open access journal of health care education, publishing peer-reviewed research, perspectives, reviews, and early documentation of new ideas and trends. Medical Education Online aims to disseminate information on the education and training of physicians and other health care professionals. Manuscripts may address any aspect of health care education and training, including, but not limited to: -Basic science education -Clinical science education -Residency education -Learning theory -Problem-based learning (PBL) -Curriculum development -Research design and statistics -Measurement and evaluation -Faculty development -Informatics/web
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信