Journal of Educational Evaluation for Health Professions最新文献

{"title":"Development and initial validation of an instrument measuring the perceived importance of core competencies for clinical nurse educators in Korea: a methodological study.","authors":"Heui-Kyeong Kwon, Junmoo Ahn, Kyungsan Seo, Seung-Hee Lee, Eunhee Jung, Yoonjung Lee","doi":"10.3352/jeehp.2026.23.28","DOIUrl":"https://doi.org/10.3352/jeehp.2026.23.28","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to develop an instrument to assess the perceived importance of core competencies for clinical nurse educators (CNEs) in Korea, defined here as hospital-based nurse educators under the Korean Nursing Act, and to provide initial evidence of its content validity, internal structure, and reliability.</p><p><strong>Methods: </strong>A preliminary 44-item pool, developed from international nurse educator frameworks and the literature, was refined by 8 experts through a 2-round modified Delphi process. A nationwide sample of 263 CNEs then rated the perceived importance of each item. Dimensionality was examined using parallel analysis and exploratory factor analysis based on polychoric correlations; the refined model was tested using confirmatory factor analysis (CFA) with the weighted least squares mean- and variance-adjusted estimator, composite reliability (CR), average variance extracted (AVE), Fornell-Larcker and heterotrait-monotrait discriminant validity, and alternative models.</p><p><strong>Results: </strong>Content validity was high (scale-level content validity index=0.986). Parallel analysis supported a 5-factor structure; 9 weak or redundant items were removed, consolidating the 8 domains into 5 factors comprising 35 items. The CFA model showed good fit (comparative fit index=0.977, Tucker-Lewis index=0.975, root mean square error of approximation=0.048, standardized root mean square residual=0.053; standardized loadings, 0.669-0.934). Internal consistency was high (Cronbach's α=0.873-0.913; categorical McDonald's ω=0.871-0.916), and convergent validity was supported (CR=0.91-0.95; AVE>0.50). Three factors were highly correlated; a second-order model fit well, and the general factor explained most of the reliable variance (ω_h=0.83).</p><p><strong>Conclusion: </strong>The 35-item instrument provided initial evidence of content validity, internal structure, and reliability for assessing the perceived importance of CNE competencies. These findings represent initial, rather than definitive, validity evidence and require cross-validation in an independent sample.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"28"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148882164","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Psychometric properties of instruments assessing the ethics of teaching: a COSMIN-based systematic review.","authors":"Kesinee Chaimo, Sureeporn Thanasilp","doi":"10.3352/jeehp.2026.23.12","DOIUrl":"10.3352/jeehp.2026.23.12","url":null,"abstract":"<p><strong>Purpose: </strong>This systematic review aimed to identify and critically evaluate instruments assessing the ethics of teaching and related moral constructs among educators, with a focus on their psychometric properties and applicability to health professions education.</p><p><strong>Methods: </strong>A systematic search was conducted in PubMed, ERIC, Scopus, and Emerald Insight databases through January 31, 2026. Only English-language studies were included. Measurement properties were evaluated using COSMIN (consensus-based standards for the selection of health measurement instruments) and COSMIN-modified GRADE (Grading of Recommendations Assessment, Development, and Evaluation) approaches.</p><p><strong>Results: </strong>Of 246 records, 6 instruments met the inclusion criteria: ESSQ (Ethical Sensitivity Scale Questionnaire), ELS (Ethical Leadership Scale), EEQ (Ethical Evaluation Questionnaire), TEPI (Teaching-Profession Ethical Principles Inventory), TCPERSS (Teachers' Compliance with Professional Ethics in Relations with Students Scale), and MCI (Moral Competency Inventory). Psychometric properties were sufficiently reported for selected domains, primarily internal consistency and structural validity (Cronbach's α=0.74-0.97). However, construct validity (hypothesis testing), test-retest reliability, and cross-cultural validation were inconsistently reported. The quality of evidence was moderate because of limited cross-context validation. Notably, no tools were specifically developed for health professions education. Most identified instruments focused on classroom pedagogy, potentially overlooking clinical instruction, bedside teaching, and workplace-based learning, where power dynamics and clinical pressures coexist. Developing tools that capture the \"ethics of the clinical encounter\" would help more accurately reflect the realities faced by health professions educators.</p><p><strong>Conclusion: </strong>Existing instruments demonstrate sufficient psychometric properties in general education but reveal critical measurement gaps for health professions education. These findings provide an empirical basis for developing context-specific instruments to improve the evaluation of ethical teaching in clinical and healthcare settings.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"12"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13338586/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148044017","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Short-term knowledge gains and learner satisfaction after a basic radiology course for family medicine residents in a 3-dimensional virtual environment: a multi-cohort pretest-posttest study in Spain.","authors":"Alberto Pino-Postigo, Rocio Lorenzo-Alvarez, Dolores Dominguez-Pinos, Eugenio Lázaro Navarro-Sanchis, Eduardo Ochando-Pulido, Miguel Jose Ruiz-Gomez, Francisco Sendra-Portero","doi":"10.3352/jeehp.2026.23.25","DOIUrl":"10.3352/jeehp.2026.23.25","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to evaluate a structured basic radiology course for Family Medicine residents delivered in a 3-dimensional (3D) virtual environment (Second Life). We hypothesized that participation would be associated with higher post-course knowledge test scores and high learner satisfaction.</p><p><strong>Methods: </strong>A quasi-experimental multi-cohort pretest-posttest study was conducted across 3 consecutive cohorts of Family Medicine residents in Spain in 2019. Ninety-six participants engaged in a 15-day course combining synchronous and asynchronous activities. Sixty-five participants provided paired pre- and post-intervention knowledge assessments, which were analyzed using paired t-tests. Learner satisfaction was evaluated using a structured questionnaire with Likert-scale items, numerical ratings (0-10), and open-ended responses. Pre- and post-test scores and ratings were compared using paired t-tests, and Mann-Whitney U tests were used for Likert-scale data. Qualitative data from open-ended responses were analyzed using thematic coding.</p><p><strong>Results: </strong>Participants had significantly higher post-course knowledge test scores, increasing from 47.4±11.6 to 58.2±11.9 (mean difference, 10.8; P<0.001; paired-sample Cohen's d [dz]=0.67). High agreement was observed across most satisfaction items, with median scores ranging from 4 to 5. Content relevance, usefulness for clinical practice, and instructor performance received the highest ratings. Lower scores were observed for peer interaction and platform usability.</p><p><strong>Conclusion: </strong>A structured radiology course delivered in a 3D virtual environment was associated with higher immediate post-course knowledge test scores and high learner satisfaction among Family Medicine residents. The findings support the feasibility of repeated course delivery, although controlled studies are warranted.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"25"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13575735/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148702510","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Developing clinical skills assessment modules for traditional, complementary, and integrative medicine in Korea: a participatory action research study.","authors":"Yoonjin Jeong, Seung Hwan Mun, Eunbyul Cho, Hye-Yoon Lee, Sang Woo Shin, Soyeon Kim, Eui-Hyoung Hwang, Man-Suk Hwang, Eunseok Kim, Jungyun Lee","doi":"10.3352/jeehp.2026.23.10","DOIUrl":"10.3352/jeehp.2026.23.10","url":null,"abstract":"<p><strong>Purpose: </strong>This study aimed to develop pilot clinical skills assessment (CSA) modules for Korean medicine-specific procedures and to examine their preliminary appropriateness, perceived necessity, and feasibility as a foundation for future licensing-related assessment development.</p><p><strong>Methods: </strong>A participatory action research framework, supplemented by qualitative interviews, was used to develop 4 CSA modules-acupuncture, Chuna manual therapy, pulse diagnosis, and constitutional diagnosis-in collaboration with expert evaluators, students, and standardized patients. The modules were implemented as formative examinations for third-year Korean medicine students, after which semi-structured interviews were conducted to obtain feedback on module content, implementation processes, and scoring procedures. Each module was also reviewed using the RUMBA checklist (Realistic, Understandable, Measurable, Behavioral, and Achievable), together with ratings of perceived necessity and feasibility for possible future use in licensing-related assessment. Interview data were analyzed inductively at the level of individual responses and then compared across modules and participant groups.</p><p><strong>Results: </strong>Qualitative analysis yielded 3 themes: content and scoring criteria, physical environment or simulators, and education or training. Participants emphasized the need to make key aspects of performance more observable, improve authenticity through simulators or task trainers, and strengthen the capacity of scoring systems to distinguish between levels of student performance. Across all modules, mean RUMBA scores were high in the understandable, behavioral, and achievable domains, whereas measurability was more problematic, especially for pulse diagnosis.</p><p><strong>Conclusion: </strong>These pilot findings clarify both the strengths and the limitations of Korean medicine-specific CSA modules. The modules received favorable ratings for understandability and achievability, whereas lower ratings for measurability and realism identified priorities for refinement before wider use. This study provides preliminary guidance for the continued development and broader evaluation of Korean medicine-specific performance assessments.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"10"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13311766/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148018123","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Virtual reality simulation with eye-tracking feedback versus mannequin-based training for situational awareness in trauma management under simulated emergency department interruptions in Iran: a pilot randomized controlled trial.","authors":"Navaz Emadi, Rita Mojtahedzadeh, Seyyed Farshad Allameh, Kamal Basiri, Aeen Mohammadi","doi":"10.3352/jeehp.2026.23.8","DOIUrl":"10.3352/jeehp.2026.23.8","url":null,"abstract":"<p><strong>Purpose: </strong>To primarily examine the feasibility of implementing eye-tracking-based feedback within a virtual reality (VR) trauma simulation with realistically simulated emergency department interruptions, and to explore preliminary changes in situational awareness (SA) (primary outcome), Advanced Trauma Life Support (ATLS) performance, and trauma-management errors (exploratory outcomes) compared with conventional mannequin-based simulation.</p><p><strong>Methods: </strong>In this pilot randomized pretest-posttest study, 35 medical interns were assigned to VR training with eye-tracking heatmap feedback (n=17) or mannequin-based training with instructor verbal feedback (n=18). SA (modified Situation Awareness Global Assessment Technique), ATLS checklist performance, and trauma-management error scores were measured before and after the intervention. Within-group changes were tested with the Wilcoxon signed-rank test, and between-group differences were compared using the Mann-Whitney U test on change scores (Δ=post-pre), with effect sizes reported as r.</p><p><strong>Results: </strong>Baseline pretest performance did not differ significantly between the groups. Both groups improved in SA and ATLS performance (all P<0.001) and reduced error scores (VR: P=0.004; mannequin: P<0.001). In exploratory between-group comparisons, the VR group showed numerically greater improvements in SA (mean change 6.59 vs. 3.11; P=0.006, r=0.46), ATLS performance (22.12 vs. 8.22; P=0.003, r=0.48), and error reduction (-9.36 vs. -3.61; P=0.005, r=0.47). Given the pilot design, these differences should be interpreted as preliminary signals.</p><p><strong>Conclusion: </strong>In this pilot study, both training modalities were associated with improved SA and ATLS performance and with fewer errors, with point estimates favoring the VR condition. These preliminary signals suggest that VR with eye-tracking feedback may be a promising option for trauma training in interruption-rich, emergency department-like settings and warrants further evaluation in larger studies.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"8"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13222740/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147784887","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Psychometric characteristics of Korean Medical Licensing Examination items from 2012 to 2022 analyzed using the Rasch model and 2-parameter logistic model.","authors":"Dong Gi Seo, Jeongwook Choi, Mee Young Kim, Sun Huh","doi":"10.3352/jeehp.2026.23.23","DOIUrl":"10.3352/jeehp.2026.23.23","url":null,"abstract":"<p><strong>Purpose: </strong>This study characterized the psychometric properties of Korean Medical Licensing Examination items administered from 2012 to 2022 using the Rasch and 2-parameter logistic models and descriptively examined items classified as highly difficult.</p><p><strong>Methods: </strong>Item parameters were estimated separately for each examination year using the Rasch and 2-parameter logistic models implemented in the irtQ package in R. Descriptive statistics and correlations between model-based difficulty estimates were calculated. Items with a 2-parameter logistic difficulty estimate of b ≥2.0 were subjected to content analysis.</p><p><strong>Results: </strong>Correlations between Rasch and 2-parameter logistic item-difficulty estimates ranged from 0.70 to 0.80 across examination years. Under the 2-parameter logistic model, the proportion of items with b ≥2.0 ranged from 4.7% to 8.9% and showed no monotonic temporal trend; the proportions were 5.0% in 2021 and 5.6% in 2022. The Rasch model generally produced smaller estimated standard errors than the 2-parameter logistic model.</p><p><strong>Conclusion: </strong>Rasch and 2-parameter logistic difficulty estimates showed strongly correlated rankings, although correlation alone did not establish agreement between their absolute estimates. The Rasch model demonstrated greater numerical stability under the present calibration conditions, but model selection should also consider model fit, test information, and the intended use of scores. Because examination years were calibrated separately and no pre-disclosure or control condition was available, the observed annual differences cannot be interpreted as evidence that item disclosure caused changes in item difficulty or discrimination.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"23"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148670835","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Non-surgical residents' experiences of relationships, identity, and burnout during residency in the United States: a qualitative study.","authors":"Eileen Reed, Kimberly Montez, Leila Hamzi DeWitt, Chad David McCalla, Andrea Triplett, Shannon Hanson, Erealda Prendaj, Elizabeth Halvorson","doi":"10.3352/jeehp.2026.23.17","DOIUrl":"10.3352/jeehp.2026.23.17","url":null,"abstract":"<p><strong>Purpose: </strong>This study explored non-surgical residents' and their domestic partners' perceptions of how residency affected their lives and relationships through semi-structured interviews.</p><p><strong>Methods: </strong>This qualitative study consisted of 10 semi-structured interviews with non-surgical residents at a single institution and 3 interviews with domestic partners. Residents from Child Neurology, Family Medicine, Internal Medicine, and Psychiatry were represented. The interviews were recorded, transcribed, and collaboratively coded using Atlas.ti ver. 22.0. Themes were identified using team-based thematic analysis.</p><p><strong>Results: </strong>Analysis of the interviews yielded 4 themes: (1) residency training results in noticeable changes in personality traits; (2) residency affects life beyond the workplace; (3) residents and their domestic partners employ strategies to support relationship success during residency; and (4) residency program interventions may support well-being.</p><p><strong>Conclusion: </strong>This study found that residency training was perceived to affect residents, domestic partners, and their relationships in both beneficial and challenging ways. Residents' domestic partners remain a valuable yet underutilized resource for learning more about the residency experience.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"17"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148260031","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Analysis of digital twin applications in nursing practice and education: a scoping review.","authors":"A Reum Lim, Hyun Kyoung Kim","doi":"10.3352/jeehp.2026.23.13","DOIUrl":"10.3352/jeehp.2026.23.13","url":null,"abstract":"<p><p>This scoping review examined research applying digital twins in nursing practice and education and summarized their application domains, methods, outcomes, and implications. A human digital twin is a virtual health replica modeled from real-world data. This study followed the 5-stage scoping review process proposed by Arksey and O'Malley. Two researchers independently conducted the literature search without restrictions on publication year. From April 1 to 15, 2026, the Cochrane Library, PubMed, Embase, CINAHL, ERIC, and RISS databases were searched, and 15 studies were ultimately included. Digital twin applications were identified in 3 major domains: clinical practice and patient-centered care, education and training, and decision-making and workflow management. Application methods and outcomes varied according to technological implementation and included (1) modeling and data-driven prediction, (2) development of immersive learning and practice-training environments, and (3) system integration and decision-support frameworks. In clinical settings, multimodal patient data can be analyzed using artificial intelligence and machine learning to generate a virtual persona resembling the patient, thereby facilitating real-time personalized nursing care and self-management. In educational settings, digital twins can provide realistic and safe learning environments that enhance training effectiveness. Digital twins show substantial potential to advance predictive and personalized nursing in both clinical practice and education. Their data-driven capabilities are expected to contribute to innovative applications in future nursing practice and educational environments.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"13"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13433139/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148206344","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Socioeconomic and economic factors affecting access and progression in medical schools: a systematic review and meta-analysis.","authors":"Arash Arianpoor, Alexia Pena, Annette Mercer, Jennifer Cox, Dimitra Lekkas, Francis Ruel Geronimo, Heidi Waldron, John Randal, Marcus Dabner, Marita Lynagh, Nalini Pather, Nicole Shepherd, Nigel Robb, Rose Berdin, Tim Wilkinson, Wendy Hu, Boaz Shulruf, Pin-Hsiang Huang","doi":"10.3352/jeehp.2026.23.6","DOIUrl":"10.3352/jeehp.2026.23.6","url":null,"abstract":"<p><strong>Purpose: </strong>Socioeconomic disadvantage remains a major determinant of equitable access to, and progression within, medical education. This systematic review and meta-analysis examines both the impact and the magnitude of financial and economic disadvantage on student selection and progression in medical school.</p><p><strong>Methods: </strong>Studies were included if they reported associations between socioeconomic indicators (e.g., parental income, occupation, education, geographic deprivation, or premedical debt) and selection or progression outcomes, and were excluded if they lacked clearly defined economic predictors or sufficient data for binary effect sizes. Searches were conducted across PubMed, Scopus, ERIC, Embase, ProQuest, and EBSCO (2005-2025). Study selection employed an active machine-learning screening process. Extracted data included sample characteristics, socioeconomic measures, and outcome types, with risk of bias assessed using the Risk of Bias Instrument. Random-effects meta-analysis was conducted where appropriate.</p><p><strong>Results: </strong>Thirty-two studies of medical programs were included, yielding 28 effect sizes for selection and 9 for progression. Household economic and educational disadvantage, identified through parental indices, was consistently associated with reduced odds of admission (odds ratio [OR], 0.6; 95% confidence interval [CI], 0.55-0.65) and poorer progression (OR, 0.56; 95% CI, 0.53-0.59). Geographic deprivation also exerted a negative effect, particularly on selection (OR, 0.69; 95% CI, 0.5-0.93).</p><p><strong>Conclusion: </strong>Socioeconomic disadvantage exerts a pervasive influence across the medical education continuum. Addressing these inequities requires sustained financial, academic, and psychosocial support both before and during their studies. Students' economic circumstances should therefore be considered in medical school selection policy and curriculum development to further enhance equity within medical schools and the profession.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"6"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13181141/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"147692798","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Refusal-aware evaluation of frontier AI models available in June 2026 using the Japanese National License Examination for Pharmacists: a comparative study.","authors":"Hiroyasu Sato, Katsuhiko Ogasawara, Hidehiko Sakurai","doi":"10.3352/jeehp.2026.23.29","DOIUrl":"10.3352/jeehp.2026.23.29","url":null,"abstract":"<p><strong>Purpose: </strong>Conventional single-run accuracy may be insufficient for evaluating frontier generative artificial intelligence (AI) models when safety-related refusals occur. This 4-model benchmark examined the need for repeated, refusal-aware evaluation using the Japanese National License Examination for Pharmacists (JNLEP).</p><p><strong>Methods: </strong>ChatGPT GPT-5.5, Gemini 3.5 Flash, Claude Opus 4.8, and Claude Fable 5 were evaluated using all 345 questions from the 107th JNLEP. The original Japanese questions, including image-containing items, were submitted through application programming interfaces (APIs) in 3 independent runs. Refusals were treated as incorrect when overall accuracy was calculated. For Fable 5, accuracy excluding refusals, refusal rate, refusal consistency across runs, the subject-wise distribution of refusals, and system-assigned refusal categories were also evaluated.</p><p><strong>Results: </strong>The mean overall accuracies were 98.7% for GPT-5.5, 98.3% for Gemini 3.5 Flash, 96.1% for Claude Opus 4.8, and 70.3% for Claude Fable 5. Fable 5 had a mean refusal rate of 29.0%, whereas its mean accuracy excluding refusals was 99.0%. Among the 345 items, 93 were refused in all 3 runs, 14 were refused inconsistently across runs, and 238 were never refused. All 300 refusal responses were assigned to the bio category. Refusals were most frequent in Biology (90.0%) and Pharmacology (69.2%) but uncommon in Practice (2.5%).</p><p><strong>Conclusion: </strong>Near-saturation benchmark performance coexisted with frequent and partly run-dependent refusals in a safeguard-equipped model. Overall accuracy, accuracy excluding refusals, refusal rate, and refusal consistency describe complementary aspects of performance; therefore, repeated, refusal-aware evaluation is needed to interpret frontier AI models in pharmacy education.</p>","PeriodicalId":46098,"journal":{"name":"Journal of Educational Evaluation for Health Professions","volume":"23 ","pages":"29"},"PeriodicalIF":5.5,"publicationDate":"2026-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148882074","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}