Measuring personal characteristics in applicants to German medical schools: Piloting an online Situational Judgement Test with an open-ended response format.

IF 1.5 Q2 EDUCATION, SCIENTIFIC DISCIPLINES

GMS Journal for Medical Education Pub Date : 2024-06-17 eCollection Date: 2024-01-01 DOI:10.3205/zma001685

Mirjana Knorr, Ina Mielke, Dorothee Amelung, Mahla Safari, Oana R Gröne, Simon M Breil, Alexander MacIntosh

{"title":"Measuring personal characteristics in applicants to German medical schools: Piloting an online Situational Judgement Test with an open-ended response format.","authors":"Mirjana Knorr, Ina Mielke, Dorothee Amelung, Mahla Safari, Oana R Gröne, Simon M Breil, Alexander MacIntosh","doi":"10.3205/zma001685","DOIUrl":null,"url":null,"abstract":"Objectives: Situational Judgement Tests (SJT) are a cost-efficient method for the assessment of personal characteristics (e.g., empathy, professionalism, ethical thinking) in medical school admission. Recently, complex open-ended response format SJTs have become more feasible to conduct. However, research on their applicability to a German context is missing. This pilot study tests the acceptability, reliability, subgroup differences, and validity of an online SJT with open-ended response format developed in Canada (\"Casper\").Methods: German medical school applicants and students from Hamburg were invited to take Casper in 2020 and 2021. The test consisted of 12 video- and text-based scenarios, each followed by three open-ended questions. Participants subsequently evaluated their test experience in an online survey. Data on sociodemographic characteristics, other admission criteria (Abitur, TMS, HAM-Nat, HAM-SJT) and study success (OSCE) was available in a central research database (stav).Results: The full sample consisted of 582 participants. Test-takers' global perception of Casper was positive. Internal consistency was satisfactory in both years (α=0.73; 0.82) while interrater agreement was moderate (ICC(1,2)=0.54). Participants who were female (d=0.37) or did not have a migration background (d=0.40) received higher scores. Casper scores correlated with HAM-SJT (r=.18) but not with OSCE communication stations performance. The test was also related to Abitur grades (r=-.15), the TMS (r=.18), and HAM-Nat logical reasoning scores (r=.23).Conclusion: This study provides positive evidence for the acceptability, internal consistency, and convergent validity of Casper. The selection and training of raters as well as the scenario content require further observation and adjustments to a German context to improve interrater reliability and predictive validity.","PeriodicalId":45850,"journal":{"name":"GMS Journal for Medical Education","volume":"41 3","pages":"Doc30"},"PeriodicalIF":1.5000,"publicationDate":"2024-06-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11310783/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"GMS Journal for Medical Education","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.3205/zma001685","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/1/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"EDUCATION, SCIENTIFIC DISCIPLINES","Score":null,"Total":0}

引用次数: 0

Abstract

Objectives: Situational Judgement Tests (SJT) are a cost-efficient method for the assessment of personal characteristics (e.g., empathy, professionalism, ethical thinking) in medical school admission. Recently, complex open-ended response format SJTs have become more feasible to conduct. However, research on their applicability to a German context is missing. This pilot study tests the acceptability, reliability, subgroup differences, and validity of an online SJT with open-ended response format developed in Canada ("Casper").

Methods: German medical school applicants and students from Hamburg were invited to take Casper in 2020 and 2021. The test consisted of 12 video- and text-based scenarios, each followed by three open-ended questions. Participants subsequently evaluated their test experience in an online survey. Data on sociodemographic characteristics, other admission criteria (Abitur, TMS, HAM-Nat, HAM-SJT) and study success (OSCE) was available in a central research database (stav).

Results: The full sample consisted of 582 participants. Test-takers' global perception of Casper was positive. Internal consistency was satisfactory in both years (α=0.73; 0.82) while interrater agreement was moderate (ICC(1,2)=0.54). Participants who were female (d=0.37) or did not have a migration background (d=0.40) received higher scores. Casper scores correlated with HAM-SJT (r=.18) but not with OSCE communication stations performance. The test was also related to Abitur grades (r=-.15), the TMS (r=.18), and HAM-Nat logical reasoning scores (r=.23).

Conclusion: This study provides positive evidence for the acceptability, internal consistency, and convergent validity of Casper. The selection and training of raters as well as the scenario content require further observation and adjustments to a German context to improve interrater reliability and predictive validity.

查看原文本刊更多论文

测量德国医学院申请人的个人特征：试行开放式在线情境判断测试。

目的：情境判断测验（SJT）是一种经济有效的方法，用于评估医学院入学考试中的个人特征（如同理心、职业精神、道德思维）。最近，复杂的开放式回答形式的 SJT 变得更加可行。然而，有关其在德国环境中适用性的研究尚属空白。本试验研究测试了加拿大开发的开放式回答格式在线 SJT（"Casper"）的可接受性、可靠性、亚组差异和有效性：方法：邀请德国医学院申请者和来自汉堡的学生在 2020 年和 2021 年参加 Casper 考试。测试包括 12 个基于视频和文本的场景，每个场景后有三个开放式问题。参与者随后通过在线调查对其测试体验进行了评估。有关社会人口学特征、其他入学标准（Abitur、TMS、HAM-Nat、HAM-SJT）和学习成功率（OSCE）的数据可从中央研究数据库（stav）中获取：全部样本包括 582 名参与者。测试者对卡斯帕的总体评价是积极的。两年的内部一致性均令人满意（α=0.73；0.82），考官间的一致性为中等（ICC(1,2)=0.54）。女性（d=0.37）或无移民背景（d=0.40）的参与者得分更高。Casper得分与HAM-SJT（r=0.18）相关，但与OSCE交流站成绩无关。该测试还与Abitur成绩（r=-.15）、TMS（r=.18）和HAM-Nat逻辑推理成绩（r=.23）相关：本研究为 Casper 的可接受性、内部一致性和收敛效度提供了积极的证据。评定者的选择和培训以及情景内容需要进一步观察，并根据德国的情况进行调整，以提高评定者之间的可靠性和预测有效性。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

GMS Journal for Medical Education EDUCATION, SCIENTIFIC DISCIPLINES-

CiteScore

3.40

自引率

12.50%

发文量

审稿时长

25 weeks

期刊介绍： GMS Journal for Medical Education (GMS J Med Educ) – formerly GMS Zeitschrift für Medizinische Ausbildung – publishes scientific articles on all aspects of undergraduate and graduate education in medicine, dentistry, veterinary medicine, pharmacy and other health professions. Research and review articles, project reports, short communications as well as discussion papers and comments may be submitted. There is a special focus on empirical studies which are methodologically sound and lead to results that are relevant beyond the respective institution, profession or country. Please feel free to submit qualitative as well as quantitative studies. We especially welcome submissions by students. It is the mission of GMS Journal for Medical Education to contribute to furthering scientific knowledge in the German-speaking countries as well as internationally and thus to foster the improvement of teaching and learning and to build an evidence base for undergraduate and graduate education. To this end, the journal has set up an editorial board with international experts. All manuscripts submitted are subjected to a clearly structured peer review process. All articles are published bilingually in English and German and are available with unrestricted open access. Thus, GMS Journal for Medical Education is available to a broad international readership. GMS Journal for Medical Education is published as an unrestricted open access journal with at least four issues per year. In addition, special issues on current topics in medical education research are also published. Until 2015 the journal was published under its German name GMS Zeitschrift für Medizinische Ausbildung. By changing its name to GMS Journal for Medical Education, we wish to underline our international mission.