Journal of Educational Evaluation for Health Professions最新文献

{"title":"Feasibility of applying computerized adaptive testing to the Clinical Medical Science Comprehensive Examination in Korea: a psychometric study.","authors":"Jeongwook Choi, Sung-Soo Jung, Eun Kwang Choi, Kyung Sik Kim, Dong Gi Seo","doi":"10.3352/jeehp.2025.22.29","DOIUrl":"10.3352/jeehp.2025.22.29","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to investigate the feasibility of transitioning the Clinical Medical Science Comprehensive Examination (CMSCE) to computerized adaptive testing (CAT) in Korea, thereby providing greater opportunities for medical students to accurately compare their clinical competencies with peers nationwide and to monitor their own progress.</p><p><strong>Methods: </strong>A medical self-assessment using CAT was conducted from March to June 2023, involving 1,541 medical students who volunteered from 40 medical colleges in Korea. An item bank consisting of 1,145 items from previously administered CMSCE examinations (2019-2021) hosted by the Medical Education Assessment Corporation was established. Items were selected through 2-stage filtering, based on classical test theory (discrimination index above 0.15) and item response theory (discrimination parameter estimates above 0.6 and difficulty parameter estimates between -5 and +5). Maximum Fisher information was employed as the item selection method, and maximum likelihood estimation was used for ability estimation.</p><p><strong>Results: </strong>The CAT was successfully administered without significant issues. The stopping rule was set at a standard error of measurement of 0.25, with a maximum of 50 items for ability estimation. The mean ability score was 0.55, with an average of 28 items administered per student. Students at extreme ability levels reached the maximum of 50 items due to the limited availability of items at appropriate difficulty levels.</p><p><strong>Conclusion: </strong>The medical self-assessment CAT, the first of its kind in Korea, was successfully implemented nationwide without significant problems. These results indicate strong potential for expanding the use of CAT in medical education assessments.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"29"},"PeriodicalIF":5.5,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12976739/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145201748","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Performance of ChatGPT-4 on the French Board of Plastic Reconstructive and Aesthetic Surgery written exam: a descriptive study.","authors":"Emma Dejean-Bouyer, Anoujat Kanlagna, François Thuau, Pierre Perrot, Ugo Lancien","doi":"10.3352/jeehp.2025.22.27","DOIUrl":"https://doi.org/10.3352/jeehp.2025.22.27","url":null,"abstract":"<p><strong>Purpose: </strong>This study aims to evaluate the performance of Chat Generative Pre-Trained Transformer 4 (ChatGPT-4) on the French Board of Plastic, Reconstructive, and Aesthetic Surgery written examination and to assess its role as a supplementary resource in helping medical students prepare for the qualification examination in plastic surgery.</p><p><strong>Methods: </strong>This descriptive study evaluated ChatGPT-4's performance on 213 items from the October 2024 French Board of Plastic, Reconstructive, and Aesthetic Surgery written examination. Responses were assessed for accuracy, logical reasoning, internal and external information use, and were categorized for fallacies by independent reviewers. Statistical analyses included chi-square tests and Fisher's exact test for significance.</p><p><strong>Results: </strong>ChatGPT-4 answered all questions across the 10 modules, achieving an overall accuracy rate of 77.5%. The model applied logical reasoning in 98.1% of the questions, utilized internal information in 94.4%, and incorporated external information in 91.1%.</p><p><strong>Conclusion: </strong>ChatGPT-4 performs satisfactorily on the French Board of Plastic, Reconstructive, and Aesthetic Surgery written examination. Its accuracy met the minimum passing standards for the exam. While responses generally align with expected knowledge, careful verification remains necessary, particularly for questions involving image interpretation. As artificial intelligence continues to evolve, ChatGPT-4 is expected to become an increasingly reliable tool for medical education. At present, it remains a valuable resource for assisting plastic surgery residents in their training.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"27"},"PeriodicalIF":3.7,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145193239","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Perceptions of faculty and medical students regarding an undergraduate research culture activity in Myanmar: a qualitative study.","authors":"Htain Lin Aung, Moe Oo Thant, July Maung Maung, Ye Hlaing Oo, Thin Thin Toe, Hla Moe","doi":"10.3352/jeehp.2025.22.33","DOIUrl":"10.3352/jeehp.2025.22.33","url":null,"abstract":"<p><strong>Purpose: </strong>This study explored the perceptions of faculty members and third-year medical students regarding the research culture activity (RCA), a program designed to engage undergraduates in research at the University of Medicine, Mandalay, Myanmar. It aimed to identify the knowledge, attitudes, and skills (KAS) gained, the challenges encountered, and suggestions for improvement.</p><p><strong>Methods: </strong>This qualitative study employed 4 semi-structured focus group discussions with 17 third-year medical students and 16 faculty members who participated in the 2020 RCA. Student responses related to KAS were analyzed using a deductive framework approach, while challenges and suggestions were examined through inductive thematic analysis. Discussions were audio-recorded, transcribed verbatim in Burmese, translated into English, and collaboratively coded using Atlas.ti version 9.0.5.</p><p><strong>Results: </strong>Participants reported improved understanding of scientific literature, greater responsibility, strengthened teamwork, and enhanced practical research skills. Reported challenges included limited research preparedness, scheduling conflicts, inconsistent supervision, financial constraints, and weak coordination with inpatient clinicians. Participants also suggested clearer guidelines, pre-research training, protected time, stronger supervision, and institutional budgetary support.</p><p><strong>Conclusion: </strong>The RCA provides substantial educational value in developing research competencies and remains a promising, potentially adaptable model for resource-limited settings. Its sustainability will depend on institutional commitment, supervisory capacity, and modest financial investment. Future research should prospectively assess KAS outcomes, compare supervision models and group sizes, evaluate digital workflows for efficiency, and conduct long-term follow-up of graduates' scholarly activities to build evidence for scalable implementation.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"33"},"PeriodicalIF":3.7,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12768548/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145379151","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Halted medical education and medical residents’ training in Korea, journal metrics, and appreciation to reviewers and volunteers","authors":"Sun Huh","doi":"10.3352/jeehp.2025.22.1","DOIUrl":"10.3352/jeehp.2025.22.1","url":null,"abstract":"","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"1"},"PeriodicalIF":5.5,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11880820/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142980326","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Decline in attrition rates in United States pediatric residency and fellowship programs, 2007-2020: a repeated cross-sectional study.","authors":"Emma Omoruyi, Greg Russell, Kimberly Montez","doi":"10.3352/jeehp.2025.22.24","DOIUrl":"10.3352/jeehp.2025.22.24","url":null,"abstract":"<p><strong>Purpose: </strong>Declining fill rates in US pediatric residency and subspecialty programs requires trainee retention. Attrition, defined as transfers, withdrawals, dismissals, unsuccessful completions, or deaths, disrupts program function and impacts the pediatric workforce pipeline. It aims to evaluate attrition trends among pediatric residents and fellows in Accreditation Council for Graduate Medical Education (ACGME)-accredited programs from 2007 to 2020.</p><p><strong>Methods: </strong>This repeated cross-sectional study analyzed publicly available ACGME Data Resource Book records. Attrition rates and 95% confidence intervals (CIs) were calculated overall and by subspecialty. Logistic regression assessed temporal changes; odds ratios (ORs) compared 2020 to 2007.</p><p><strong>Results: </strong>From 2007-2020, pediatric residents increased from 8,145 to 9,419 and fellows from 2,875 to 4,279. Aggregate annual resident attrition averaged 1.71% (range, 0.93%-2.64%), and fellow attrition ranged from 12.39%-30.87%. Transfer rates declined from 18.05 to 5.20 per 1,000 trainees (P<0.0001), withdrawals from 5.65 to 2.76 (P=0.030), and dismissals from 3.14 in 2010 to 1.27 in 2020 (P=0.0068). Odds of unsuccessful completion significantly decreased in categorical pediatrics (OR, 0.41; 95% CI, 0.29-0.58), pediatric cardiology (OR, 0.08; 95% CI, 0.01-0.64), pediatric critical care (OR, 0.14; 95% CI, 0.06-0.35), and neonatal-perinatal medicine (OR, 0.46; 95% CI, 0.20-1.08).</p><p><strong>Conclusion: </strong>Although attrition has improved, premature trainee loss can still disrupt program operations and threaten workforce development. Attrition may reflect educational environment quality, support structures, or selection processes. Greater data transparency is needed to understand demographic trends and inform equitable retention strategies, ultimately strengthening training programs and sustaining the United States pediatric workforce.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"24"},"PeriodicalIF":3.7,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12676133/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145151426","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Performance of large language models in medical licensing examinations: a systematic review and meta-analysis.","authors":"Haniyeh Nouri, Abdollah Mahdavi, Ali Abedi, Alireza Mohammadnia, Mahnaz Hamedan, Masoud Amanzadeh","doi":"10.3352/jeehp.2025.22.36","DOIUrl":"10.3352/jeehp.2025.22.36","url":null,"abstract":"<p><strong>Purpose: </strong>This study systematically evaluates and compares the performance of large language models (LLMs) in answering medical licensing examination questions. By conducting subgroup analyses based on language, question format, and model type, this meta-analysis aims to provide a comprehensive overview of LLM capabilities in medical education and clinical decision-making.</p><p><strong>Methods: </strong>This systematic review, registered in PROSPERO and following PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines, searched MEDLINE (PubMed), Scopus, and Web of Science for relevant articles published up to February 1, 2025. The search strategy included Medical Subject Headings (MeSH) terms and keywords related to (\"ChatGPT\" OR \"GPT\" OR \"LLM variants\") AND (\"medical licensing exam*\" OR \"medical exam*\" OR \"medical education\" OR \"radiology exam*\"). Eligible studies evaluated LLM accuracy on medical licensing examination questions. Pooled accuracy was estimated using a random-effects model, with subgroup analyses by LLM type, language, and question format. Publication bias was assessed using Egger's regression test.</p><p><strong>Results: </strong>This systematic review identified 2,404 studies. After removing duplicates and excluding irrelevant articles through title and abstract screening, 36 studies were included after full-text review. The pooled accuracy was 72% (95% confidence interval, 70.0% to 75.0%) with high heterogeneity (I2=99%, P<0.001). Among LLMs, GPT-4 achieved the highest accuracy (81%), followed by Bing (79%), Claude (74%), Gemini/Bard (70%), and GPT-3.5 (60%) (P=0.001). Performance differences across languages (range, 62% in Polish to 77% in German) were not statistically significant (P=0.170).</p><p><strong>Conclusion: </strong>LLMs, particularly GPT-4, can match or exceed medical students' examination performance and may serve as supportive educational tools. However, due to variability and the risk of errors, they should be used cautiously as complements rather than replacements for traditional learning methods.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"36"},"PeriodicalIF":5.5,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12976628/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145542995","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Correlation between a motion analysis method and Global Operative Assessment of Laparoscopic Skills for assessing interns' performance in a simulated peg transfer task in Jordan: a validation study","authors":"Esraa Saleh Abdelall, Shadi Mohammad Hamouri, Abdallah Fawaz Al Dwairi, Omar Mefleh Al-Araidah","doi":"10.3352/jeehp.2025.22.10","DOIUrl":"10.3352/jeehp.2025.22.10","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to validate the use of ProAnalyst (Xcitex Inc.), a program for professional motion analysts to assess the performance of surgical interns while performing the peg transfer task in a simulator box for safe practice in real minimally invasive surgery.</p><p><strong>Methods: </strong>A correlation study was conducted in a multidisciplinary skills simulation lab at the Faculty of Medicine, Jordan University of Science and Technology from October 2019 to February 2020. Forty-one interns (i.e., novices and intermediates) were recruited, and an expert surgeon participated as a reference benchmark. Videos of participants’ performance were analyzed using ProAnalyst and the Global Operative Assessment of Laparoscopic Skills (GOALS). The two sets of results were analyzed to identify correlations.</p><p><strong>Results: </strong>The motion analysis scores from Proanalyst were correlated with those from GOALS for efficiency (r=+0.38, P<0.05), autonomy (r=+0.63, P<0.01), depth perception (r=+0.43, P<0.05), dexterity (r=+0.71, P<0.001), and operation flow (r=+0.88, P<0.001). Both assessment methods differentiated the participants’ performance based on their experience level.</p><p><strong>Conclusion: </strong>The motion analysis scoring method using Proanalyst provides an objective, time-efficient, and reproducible assessment of interns’ performance, with results comparable to those obtained using GOALS. It may require initial training and set-up; however, it eliminates the need for expert surgeon judgment.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"10"},"PeriodicalIF":9.3,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12012728/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143568494","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Simulation-based teaching versus traditional small group teaching for first-year medical students among high and low scorers in respiratory physiology, India: a randomized controlled trial.","authors":"Nalini Yelahanka Channegowda, Dinker Ramanand Pai, Shivasakthy Manivasakan","doi":"10.3352/jeehp.2025.22.8","DOIUrl":"https://doi.org/10.3352/jeehp.2025.22.8","url":null,"abstract":"<p><strong>Purpose: </strong>Although it is widely utilized in clinical subjects for skill training, using simulation-based education (SBE) for teaching basic science concepts to phase I medical students or pre-clinical students is limited. Simulation-based education/teaching is preferred in cardiovascular and respiratory physiology when compared to other systems because it is easy to recreate both the normal physiological component and alterations in the simulated environment, thus a promoting deep understanding of the core concepts.</p><p><strong>Methods: </strong>A block randomized study was conducted among 107 phase 1 (first-year) medical undergraduate students at a Deemed to be University in India. Group A received SBE and Group B traditional small group teaching. The effectiveness of the teaching intervention was assessed using pre- and post-tests. Student feedback was obtained through a self administered structured questionnaire via an anonymous online survey and by in-depth interview.</p><p><strong>Results: </strong>The intervention group showed a statistically significant improvement in post-test scores compared to the control group. A sub-analysis revealed that high scorers performed better than low scorers in both groups, but the knowledge gain among low scorers was more significant in the intervention group.</p><p><strong>Conclusion: </strong>This teaching strategy offers a valuable supplement to traditional methods, fostering a deeper comprehension of clinical concepts from the outset of medical training.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"8"},"PeriodicalIF":9.3,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12012709/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"144006065","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Presidential address 2025: expansion of computer-based testing from 12 to 27 health professions by 2027 and adoption of a large language model for item generation","authors":"Hyunjoo Pai","doi":"10.3352/jeehp.2025.22.7","DOIUrl":"10.3352/jeehp.2025.22.7","url":null,"abstract":"","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"7"},"PeriodicalIF":9.3,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11934035/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143013023","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Performance of GPT-4o and o1-Pro on United Kingdom Medical Licensing Assessment-style items: a comparative study.","authors":"Behrad Vakili, Aadam Ahmad, Mahsa Zolfaghari","doi":"10.3352/jeehp.2025.22.30","DOIUrl":"10.3352/jeehp.2025.22.30","url":null,"abstract":"<p><strong>Purpose: </strong>Large language models (LLMs) such as ChatGPT, and their potential to support autonomous learning for licensing exams like the UK Medical Licensing Assessment (UKMLA), are of growing interest. However, empirical evaluations of artificial intelligence (AI) performance against the UKMLA standard remain limited.</p><p><strong>Methods: </strong>We evaluated the performance of 2 recent ChatGPT versions, GPT-4o and o1-Pro, on a curated set of 374 UKMLA-style single-best-answer items spanning diverse medical specialties. Statistical comparisons using McNemar's test assessed the significance of differences between the 2 models. Specialties were analyzed to identify domain-specific variation. In addition, 20 image-based items were evaluated.</p><p><strong>Results: </strong>GPT-4o achieved an accuracy of 88.8%, while o1-Pro achieved 93.0%. McNemar's test revealed a statistically significant difference in favor of o1-Pro. Across specialties, both models demonstrated excellent performance in surgery, psychiatry, and infectious diseases. Notable differences arose in dermatology, respiratory medicine, and imaging, where o1-Pro consistently outperformed GPT-4o. Nevertheless, isolated weaknesses in general practice were observed. The analysis of image-based items showed 75% accuracy for GPT-4o and 90% for o1-Pro (P=0.25).</p><p><strong>Conclusion: </strong>ChatGPT shows strong potential as an adjunct learning tool for UKMLA preparation, with both models achieving scores above the calculated pass mark. This underscores the promise of advanced AI models in medical education. However, specialty-specific inconsistencies suggest AI tools should complement, rather than replace, traditional study methods.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"22 ","pages":"30"},"PeriodicalIF":3.7,"publicationDate":"2025-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12688319/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"145259626","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}