The equivalence of a high-stakes objective structured clinical exam adapted to suit a virtual delivery format

IF 2.1 4区 医学 Q3 HEALTH CARE SCIENCES & SERVICES
Karen Coetzee MA, Luxshi Amirthalingam BSc, Tabasom Eftekari BComm, Sandra Monteiro PhD
{"title":"The equivalence of a high-stakes objective structured clinical exam adapted to suit a virtual delivery format","authors":"Karen Coetzee MA,&nbsp;Luxshi Amirthalingam BSc,&nbsp;Tabasom Eftekari BComm,&nbsp;Sandra Monteiro PhD","doi":"10.1111/jep.14167","DOIUrl":null,"url":null,"abstract":"<div>\n \n \n <section>\n \n <h3> Introduction</h3>\n \n <p>The COVID-19 pandemic necessitated rapid adaptation of clinical competence assessments, including the transition of Objective Structured Clinical Examinations (OSCE) from in-person to virtual formats. This study investigates the construct equivalence of a high-stakes OSCE, originally designed for in-person delivery, when adapted for a virtual format.</p>\n </section>\n \n <section>\n \n <h3> Methods</h3>\n \n <p>A retrospective analysis was conducted using OSCE scores from the Internationally Educated Nurse Competency Assessment Program (IENCAP®). Data were collected from 15 exam administrations between January 2018 and June 2022, encompassing 2021 examinees (1936 in-person, 85 virtual). The Many-Facet Rasch Measurement (MFRM) model was employed to analyze the invariance of examinee ability, case difficulty, and criteria difficulty across in-person and virtual formats.</p>\n </section>\n \n <section>\n \n <h3> Results</h3>\n \n <p>Results revealed overall examinee ability estimates remained invariant regardless of the OSCE format, while invariant violations were identified in only three of the 15 cases (<i>N</i> = 20%) adapted to suit the virtual format. The most significant adaptation, namely the use of a verbal physical examination to suit the virtual context achieved equivalence to its hands-on in-person counterpart given evidence of invariance across criteria estimates. Interestingly, criteria scores in invariant violated cases displayed a higher level of stability or consistency across the virtual OSCE formats versus their in-person counterpart highlighting a potential benefit of the virtual versus in-person format and potentially linked to the verbal physical examination.</p>\n </section>\n \n <section>\n \n <h3> Conclusion</h3>\n \n <p>The study found that while examinee ability and case difficulty estimates exhibited some invariance between in-person and virtual OSCE formats, criteria involving physical assessments faced challenges in maintaining construct equivalence. These findings highlight the need for careful consideration in adapting high-stakes clinical assessments to virtual formats to ensure fairness and reliability.</p>\n </section>\n </div>","PeriodicalId":15997,"journal":{"name":"Journal of evaluation in clinical practice","volume":"31 1","pages":""},"PeriodicalIF":2.1000,"publicationDate":"2024-10-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://onlinelibrary.wiley.com/doi/epdf/10.1111/jep.14167","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of evaluation in clinical practice","FirstCategoryId":"3","ListUrlMain":"https://onlinelibrary.wiley.com/doi/10.1111/jep.14167","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"HEALTH CARE SCIENCES & SERVICES","Score":null,"Total":0}
引用次数: 0

Abstract

Introduction

The COVID-19 pandemic necessitated rapid adaptation of clinical competence assessments, including the transition of Objective Structured Clinical Examinations (OSCE) from in-person to virtual formats. This study investigates the construct equivalence of a high-stakes OSCE, originally designed for in-person delivery, when adapted for a virtual format.

Methods

A retrospective analysis was conducted using OSCE scores from the Internationally Educated Nurse Competency Assessment Program (IENCAP®). Data were collected from 15 exam administrations between January 2018 and June 2022, encompassing 2021 examinees (1936 in-person, 85 virtual). The Many-Facet Rasch Measurement (MFRM) model was employed to analyze the invariance of examinee ability, case difficulty, and criteria difficulty across in-person and virtual formats.

Results

Results revealed overall examinee ability estimates remained invariant regardless of the OSCE format, while invariant violations were identified in only three of the 15 cases (N = 20%) adapted to suit the virtual format. The most significant adaptation, namely the use of a verbal physical examination to suit the virtual context achieved equivalence to its hands-on in-person counterpart given evidence of invariance across criteria estimates. Interestingly, criteria scores in invariant violated cases displayed a higher level of stability or consistency across the virtual OSCE formats versus their in-person counterpart highlighting a potential benefit of the virtual versus in-person format and potentially linked to the verbal physical examination.

Conclusion

The study found that while examinee ability and case difficulty estimates exhibited some invariance between in-person and virtual OSCE formats, criteria involving physical assessments faced challenges in maintaining construct equivalence. These findings highlight the need for careful consideration in adapting high-stakes clinical assessments to virtual formats to ensure fairness and reliability.

Abstract Image

为适应虚拟授课形式而调整的高风险客观结构化临床考试的等效性。
导言:COVID-19大流行使得临床能力评估需要快速调整,包括将客观结构化临床考试(OSCE)从面对面形式过渡到虚拟形式。本研究调查了高风险 OSCE 的结构等效性,该 OSCE 原本是为面授形式设计的,在改编为虚拟形式后,其结构等效性得到了提高:方法:使用国际教育护士能力评估项目(IENCAP®)的 OSCE 分数进行了回顾性分析。数据收集自2018年1月至2022年6月期间的15次考试,共涉及2021名考生(1936名现场考生,85名虚拟考生)。采用多面拉施测量(MFRM)模型分析了考生能力、案例难度和标准难度在面对面和虚拟形式中的不变性:结果显示,无论采用哪种OSCE形式,考生的总体能力估计值都保持不变,而在15个为适应虚拟形式而调整的案例中,只有3个案例(N = 20%)发现了违反不变性的情况。最重要的改编,即使用口头体格检查来适应虚拟环境,在标准估计值不变的情况下,实现了与亲身实践的等效。有趣的是,在不同的虚拟 OSCE 形式下,不变的违规案例中的标准分数显示出较高的稳定性或一致性,这突出了虚拟 OSCE 形式相对于现场 OSCE 形式的潜在优势,并可能与口头体格检查有关:研究发现,虽然考生能力和病例难度估计在面对面和虚拟 OSCE 形式之间表现出一定的不变性,但涉及身体评估的标准在保持构造等效性方面面临挑战。这些发现突出表明,在将高风险的临床评估调整为虚拟形式时,需要慎重考虑,以确保公平性和可靠性。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
CiteScore
4.80
自引率
4.20%
发文量
143
审稿时长
3-8 weeks
期刊介绍: The Journal of Evaluation in Clinical Practice aims to promote the evaluation and development of clinical practice across medicine, nursing and the allied health professions. All aspects of health services research and public health policy analysis and debate are of interest to the Journal whether studied from a population-based or individual patient-centred perspective. Of particular interest to the Journal are submissions on all aspects of clinical effectiveness and efficiency including evidence-based medicine, clinical practice guidelines, clinical decision making, clinical services organisation, implementation and delivery, health economic evaluation, health process and outcome measurement and new or improved methods (conceptual and statistical) for systematic inquiry into clinical practice. Papers may take a classical quantitative or qualitative approach to investigation (or may utilise both techniques) or may take the form of learned essays, structured/systematic reviews and critiques.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信