JMIR Medical Education最新文献

{"title":"Development and Educational Evaluation of a Radiation Visualization System in a Virtual Reality Training Module for Radiopharmaceutical Administration: Mixed Methods Study of Students' Learning Experiences.","authors":"Akihiro Kakimoto, Daisuke Fujise, Shota Watanabe, Yutaka Otaka, Sakura Minemoto, Ayaka Yasumatsu, Shin Hasegawa","doi":"10.2196/104397","DOIUrl":"10.2196/104397","url":null,"abstract":"<p><strong>Background: </strong>Our research group previously developed a virtual reality (VR) training module for radiopharmaceutical administration, focusing on procedural skills and patient interactions. The module's educational effectiveness was validated with objective measures, such as electroencephalography and mood assessments.</p><p><strong>Objective: </strong>The authors aimed to extend the module by integrating a radiation visualization system that renders otherwise invisible radiation fields within an immersive VR environment, and to compare students' learning experiences associated with the module with those associated with conventional hands-on training.</p><p><strong>Methods: </strong>Twenty students enrolled in a radiological technician training program participated in this study. Students' learning experiences were evaluated using Likert-scale-based questionnaires, together with qualitative analysis of open-ended responses.</p><p><strong>Results: </strong>VR-based training was associated with significantly higher ratings for students' conceptual understanding of radiation behavior, radiation safety awareness, learning satisfaction, and self-assessed competency than conventional training. Conversely, perceived realism and immersion were rated significantly higher following conventional training. No significant differences were observed for operational understanding, technical skills, or system usability.</p><p><strong>Conclusions: </strong>These findings suggest that the developed VR module offers a safe and repeatable training platform that may support positively perceived learning experiences in radiopharmaceutical administration and radiation safety education.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e104397"},"PeriodicalIF":13.9,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13625173/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149082073","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"AI Perceptions, Professional Identity, and AI-Supported Clinical Decisions Among Medical Students and Clinicians: Cross-Sectional Survey and Quasi-Randomized Vignette Study.","authors":"Julia-Astrid Moldt, Teresa Festl-Wietek, Kay Nieselt, Susanne Zabel, Manfred Claassen, Samuel Wagner, Ulrike Keim, Anne Herrmann-Werner","doi":"10.2196/104151","DOIUrl":"10.2196/104151","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;The integration of AI into health care is increasingly shaping clinical practice and decision-making. Beyond technical performance, AI has implications for professional roles, clinical reasoning, and responsibility. Understanding how medical students and clinicians perceive AI, how these perceptions relate to professional identity concerns, and how clinicians evaluate AI-supported decisions in different clinical contexts is therefore important for both implementation and medical education.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to examine (1) how medical students and clinicians perceive the impact of AI on medical practice and professional identity and (2) how clinicians evaluate explainability, trustworthiness, and responsibility in AI-supported clinical decision-making across different contexts.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;A 2-phase quantitative study was conducted. In phase 1, medical students (n=93) and clinicians (n=65; N=158) completed a cross-sectional online survey assessing expectations regarding AI, perceived importance of AI across medical domains, and perceived professional identity threat. In phase 2, a separate sample of clinicians (N=68) was quasi-randomly assigned to 1 of 3 clinical vignettes (Watson, Triage, and OncoGuide) and evaluated AI-supported decisions along the dimensions of explainability, trustworthiness, and responsibility.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;In phase 1, participants reported generally positive expectations regarding factual-level impacts of AI on medical practice. However, these expectations were not significantly associated with perceived professional identity threat. Medical students reported significantly higher identity threat than clinicians, despite largely similar expectations regarding AI's factual and social impact. Perceived importance of AI across medical domains was associated with more positive expectations toward AI-supported medical practice, but not with identity threat. In phase 2, clinicians' evaluations of AI-supported decisions varied across contexts, with explainability ratings differing significantly across vignette scenarios (P=.007), with lower ratings in the Triage vignette than in the Watson and OncoGuide vignettes. Trustworthiness and responsibility did not differ significantly across scenarios.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;The findings suggest that positive expectations regarding AI may coexist with professional identity concerns, while identity threat differed between medical students and clinicians in the present sample. At the same time, explainability evaluations differed across clinical decision contexts, whereas trustworthiness and responsibility remained comparatively stable. Together, these findings point to both professional identity-related and contextual considerations in AI implementation that may not be captured by expectations regarding technological benefits alone. Considering these dimensions may be relevant","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e104151"},"PeriodicalIF":13.9,"publicationDate":"2026-09-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13669706/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149115650","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Refining the Immersive Technology Evaluation Measure for Virtual Reality Clinical Skills Among Native Arabic-Speaking Medical Students: Qualitative Study.","authors":"Afaf Sulaiman Alblooshi, Falah Mohammed Almarzooqi, Taleb M Almansoori, Anmol Punn, Marwa Gaffar Alameen, Alexander Kieu, Faten Abdullah AlRadini","doi":"10.2196/95904","DOIUrl":"10.2196/95904","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Extended reality technologies, including virtual reality (VR), augmented reality, and mixed reality, are increasingly used in medical education to create immersive and interactive learning environments. As these modalities expand, validated instruments are needed to measure learners' experiences accurately. The Immersive Technology Evaluation Measure (ITEM) is a multidomain questionnaire assessing immersion, motivation, cognitive load, usability, and debriefing. Although cognitive interviewing informed its original development, less is known about how ITEM questions function when used in a different linguistic and educational context.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to evaluate the clarity, comprehensibility, and response processes of ITEM among native Arabic-speaking medical students enrolled in an English-medium medical program. We also sought to identify linguistic, referential, and contextual sources of comprehension difficulty and use these findings to inform proposed item-level refinements.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;We conducted a qualitative cognitive interviewing study with 10 third-year and fourth-year medical students at United Arab Emirates University following VR-based clinical skills activities. Using concurrent think-aloud and verbal probing techniques, participants explained how they interpreted each ITEM question and arrived at their responses. Interview transcripts, recordings, and interviewer notes were independently reviewed using a descriptive, item-focused approach. Participant feedback was examined for recurring comprehension problems, and proposed item-level decisions were reviewed through research team consensus. Item-level saturation was reached after 8 interviews and confirmed with 2 additional interviews.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;Participants identified comprehension or response-process problems in 47.5% (19/40) of ITEM questions. Difficulties occurred across all 5 domains and were most frequent in immersion (6/9, 66.7%) and usability (6/10, 60%), followed by debriefing (4/5, 80%), motivation (2/10, 20%), and cognitive load (1/6, 16.7%). Common difficulties involved nonspecific references such as \"activity\" and \"technology,\" unfamiliar or abstract terminology, negative wording, and unclear temporal or contextual framing. For example, \"concern\" was interpreted by some participants as worry rather than focus, and \"mentally demanding\" was interpreted in relation to mental health rather than cognitive effort. Cognitive interviewing informed different item-level decisions rather than a uniform revision: 11 problematic questions received proposed wording or contextual revisions, 2 received presentation-only modifications, and 6 were retained unchanged where clarification could introduce a meaning not clearly established in the original item. Six additional questions received limited contextual or terminology-standardizing changes for consistency. No items were re","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e95904"},"PeriodicalIF":13.9,"publicationDate":"2026-09-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13618990/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149076287","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Correction: Attitudes Toward Digital Planetary Health Before and After a Lecture Series on Environmental Awareness and Future Visions: Repeated Cross-Sectional Survey Study Regarding Twin Transformation.","authors":"Julia Nitsche, Theresa Sophie Busse, Sonja Schmalen, Ina Otte, Jan Peter Ehlers","doi":"10.2196/111087","DOIUrl":"10.2196/111087","url":null,"abstract":"","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e111087"},"PeriodicalIF":13.9,"publicationDate":"2026-09-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13614621/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149065035","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Rethinking Medical Spanish Competence: Language Proficiency, Clinical Communication, and Inclusion.","authors":"Juli McCarroll","doi":"10.2196/110302","DOIUrl":"10.2196/110302","url":null,"abstract":"","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e110302"},"PeriodicalIF":13.9,"publicationDate":"2026-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13606328/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149041007","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Author's Reply: Rethinking Medical Spanish Competence: Language Proficiency, Clinical Communication, and Inclusion.","authors":"Alexandra Lopez Vera","doi":"10.2196/110709","DOIUrl":"10.2196/110709","url":null,"abstract":"","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e110709"},"PeriodicalIF":13.9,"publicationDate":"2026-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13606343/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149041026","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Medical AI Use Intention Among Medical Students and Faculty and Its Associations With AI Literacy, Perceived Benefits, and Risks: Cross-Sectional Survey Study.","authors":"Bomyee Lee, Gwanwook Bang, Su Jin Chae","doi":"10.2196/105032","DOIUrl":"10.2196/105032","url":null,"abstract":"<p><strong>Background: </strong>AI is being increasingly integrated into health care and medical education. Although AI literacy is considered an essential competency for future physicians and educators, limited evidence exists regarding how perceived benefits, perceived risks, and AI literacy jointly influence AI use intention among different learner groups.</p><p><strong>Objective: </strong>This study examined the factors associated with AI use intention among medical students and faculty members and investigated whether the relationships between AI literacy and AI use intention differed according to learner group.</p><p><strong>Methods: </strong>A cross-sectional self-report survey was conducted among 141 medical students and 94 faculty members (N=235) at a single South Korean medical school. Group-specific linear regression models were estimated for descriptive purposes. To directly test whether associations differed between groups, a pooled regression model included group-by-predictor interaction terms for perceived benefits, perceived risks, AI literacy, and AI use frequency. HC3 heteroscedasticity-robust SEs were used for primary inference, and ordinal logistic regression was conducted as a sensitivity analysis because the 5-point intention outcome showed a marked ceiling effect.</p><p><strong>Results: </strong>Perceived benefits were positively associated with AI use intention in both groups. In the pooled HC3-robust model, the group-by-AI literacy interaction was significant (<i>B</i>=0.361, 95% CI 0.083-0.638; <i>P</i>=.01), whereas the interactions for perceived benefits, perceived risks, and AI use frequency were not significant. The group-by-AI literacy interaction remained significant in ordinal logistic regression (odds ratio 2.48, 95% CI 1.24-4.98; <i>P</i>=.01). In subgroup models, the negative adjusted AI literacy coefficient among students was model dependent, and the positive perceived risk coefficient was not robust to HC3 inference. Among faculty members, AI use frequency remained positively associated with use intention using both HC3 linear regression (<i>B</i>=0.137, 95% CI 0.025-0.249; <i>P</i>=.02) and ordinal logistic regression (<i>P</i>=.02).</p><p><strong>Conclusions: </strong>Perceived benefits were consistently associated with stronger AI use intention. The adjusted association between AI literacy and use intention differed between students and faculty, although the subgroup-specific student coefficient was sensitive to model specification. These findings provide exploratory evidence that learner context may shape the relationship between AI literacy and broad self-reported AI use intention; they do not establish causal effects or the effectiveness of specific curricular strategies.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e105032"},"PeriodicalIF":13.9,"publicationDate":"2026-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13606327/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149040995","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Evaluating Large Language Model-Based Automated Scoring in a Voice-Based Virtual Standardized Patient Platform for Medical Students: Cross-Sectional Agreement Study.","authors":"Xiaoxing Gao, Xiaoming Huang, Rongrong Hu, Li Zhang, Huiting Liu, Bingqing Zhang, Chong Wei, Wei Qiu, Mengyu Zhang, Xuefeng Sun","doi":"10.2196/95578","DOIUrl":"10.2196/95578","url":null,"abstract":"<p><strong>Background: </strong>Large language model (LLM)-powered virtual standardized patients (VSPs) enable scalable clinical skills practice, but the validity of AI-generated scores relative to faculty ratings remains unclear.</p><p><strong>Objective: </strong>This study aimed to assess agreement between LLM-generated and faculty ratings of history-taking and communication performance and to examine the influence of rater and case heterogeneity.</p><p><strong>Methods: </strong>In this cross-sectional study, 92 fourth-year medical students completed one of three 15-minute voice-based VSP cases (fever, diarrhea, and cough). Ten blinded faculty raters scored performance (0-100 points total; 0-50 points per domain). AI scores were generated by DeepSeek-V3 using a calibrated prompt. Agreement was evaluated using mixed-effects models, intraclass correlation coefficients (ICC [2,1]), Spearman correlations, mean absolute error (MAE), Bland-Altman analysis, and variance partition coefficients (VPC).</p><p><strong>Results: </strong>Median total scores were similar for AI and faculty (median 93.0, IQR 89.0-95.0 vs median 94.0, IQR 91.0-95.0). Rater variability accounted for 37% of residual variance in faculty total scores (VPC=0.37). AI total scores were positively associated with faculty total scores (β=0.37, 95% CI 0.26-0.48; <i>P</i><.001; Spearman ρ=0.50, 95% CI 0.34-0.65). Absolute agreement was moderate (ICC[2,1]=0.51, 95% CI 0.34-0.65), with MAE of 3.11 points. Mixed-effects Bland-Altman analysis showed a small, not statistically significant mean bias (1.26 points, 95% CI -0.48 to 3.01; <i>P</i>=.16) and 95% limits of agreement from -4.95 to 7.48 (width=12.43 points), with proportional bias (β_proportional bias=-0.55; <i>P</i><.001). Agreement was stronger for information gathering (β=0.46; ρ=0.49; ICC=0.54; VPC=0.23) than for communication (β=0.27; ρ=0.28; ICC=0.29; VPC=0.52). A sensitivity analysis in the lowest quartile showed attenuated but consistent agreement (ICC=0.38).</p><p><strong>Conclusions: </strong>LLM-based scoring in a VSP showed moderate agreement with faculty ratings, performing better for information gathering than for communication. Due to rater and case heterogeneity, ceiling effects, and proportional bias, this method is suitable for formative use and enhanced sampling in programmatic assessment but not for independent, high-stakes summative decisions.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e95578"},"PeriodicalIF":13.9,"publicationDate":"2026-09-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13606323/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149040976","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Generative AI-Assisted Progressive-Disclosure Case-Based Learning for Clinical Reasoning in Occupational Medicine: Quasi-Experimental Study.","authors":"Peng Su, Min Hu, Chengzhi Chen, Shangcheng Xu","doi":"10.2196/103584","DOIUrl":"10.2196/103584","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Case-based learning (CBL) promotes transfer of knowledge to practice, yet occupational health CBL must also develop exposure assessment and epidemiologic thinking. Static, single-session cases that disclose all information upfront can truncate iterative reasoning and encourage premature diagnostic closure. Generative AI (GenAI) can support the efficient development of high-fidelity, progressively disclosed cases, but hallucination risks require strict quality control.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to develop and evaluate a multicomponent GenAI-assisted, 4-act progressive-disclosure CBL package for an occupational lead poisoning module, using a human-in-the-loop workflow to mitigate hallucination risk.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;In a nonrandomized posttest controlled quasi-experimental study, 224 undergraduates were assigned by administrative class to an intervention group (n=114) or a control group (n=110). The control group received conventional static CBL; the intervention group received a GenAI-assisted progressive-disclosure CBL package. The primary outcome was the standardized individual case-analysis assignment score. Secondary outcomes were the delayed final examination score, 5 self-reported learning experience domains, and 3 video-derived behavioral engagement indicators. Subgroup analyses by academic major and group-by-major interaction tests were exploratory.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;Baseline characteristics were comparable between groups. The intervention group scored higher on case analysis (mean 88.41, SD 3.85 vs mean 82.51, SD 3.94; P&lt;.001; Cohen d=1.51) and on the delayed final exam (mean 80.96, SD 8.47 vs mean 78.62, SD 6.63; P=.02; Cohen d=0.31). Significant improvements were observed in information gathering, hypothesis generation, and differential diagnosis (all P&lt;.001 after Holm correction), while treatment/management planning did not differ (P=.19). The intervention group reported higher perceived difficulty, perceived improvement in clinical reasoning, engagement, transfer of self-efficacy, and evaluation of the course materials (all P&lt;.001). Voluntary responses (mean 2.30, SD 1.11 vs mean 1.68, SD 0.92; P=.008), evidence-referencing statements (mean 1.90, SD 0.88 vs mean 1.45, SD 0.71; P=.01), and net group discussion time (mean 36.50, SD 6.88 vs mean 28.42, SD 7.02 minutes; P&lt;.001) were all higher in the intervention group. The group-by-major interaction for the final examination was not statistically significant (P=.41). Audit logs showed that all AI-generated case drafts required expert correction.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;A multicomponent GenAI-assisted, 4-act progressive-disclosure CBL package implemented with rigorous human-in-the-loop verification was associated with higher case-analysis performance, modestly higher delayed examination performance, and greater behavioral engagement in one nonrandomized cohort. Randomized and class-level mult","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e103584"},"PeriodicalIF":13.9,"publicationDate":"2026-09-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13643254/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149026266","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Large Language Model Performance on Multistep Clinical Cases: Comparative Study Across Question and Case Levels.","authors":"Jiaxue Cha, Yan Zhao, Hui Zong","doi":"10.2196/95342","DOIUrl":"10.2196/95342","url":null,"abstract":"<p><strong>Background: </strong>Most large language models (LLMs) have achieved passing scores on medical licensing examinations. However, most evaluations focus on single-question accuracy, overlooking performance on multistep patient management scenarios, such as making a diagnosis followed by a treatment plan. It is unclear if LLMs can maintain high performance on complete clinical cases.</p><p><strong>Objective: </strong>This study aimed to evaluate the performance of LLMs on multistep clinical cases and to investigate the impact of model size scaling and case complexity on performance stability.</p><p><strong>Methods: </strong>We curated a dataset of 189 unique clinical cases comprising 473 individual questions from the Chinese National Medical Licensing Examination. The number of questions related to each case ranged from 2 to 4. A physician with more than 8 years of clinical experience annotated the cases, confirming that 82.5% (156/189) contained sequentially dependent questions. We tested 4 representative LLMs: DeepSeek-R1, GPT-4o, Gemini-2.5-Flash, and Qwen2.5. We measured 2 metrics: the question pass rate (QPR) and the case pass rate (CPR), where a case was considered correct only if the model answered all its questions correctly. The consistency gap was calculated as the difference between QPR and CPR. Additionally, we tested Qwen2.5 at different parameter sizes (3B, 7B, 14B, and 32B) to examine the effect of model size. We also calculated the expected CPR and used the McNemar test with Bonferroni correction to compare model performance.</p><p><strong>Results: </strong>All LLMs achieved a QPR exceeding 83%. However, the CPR was lower for all models. DeepSeek-R1 achieved the highest QPR at 89.9% (425/473), with a CPR of 79.9% (151/189), corresponding to the smallest consistency gap at 10%. In contrast, GPT-4o exhibited a QPR of 83.5% (395/473) and a CPR of 65.6% (124/189), resulting in the largest consistency gap at 17.9%. DeepSeek-R1 and Qwen2.5-32B significantly outperformed GPT-4o in both QPR and CPR (P<.05). Observed and expected CPR values were closely aligned across models. Increasing the model size from 3B to 32B reduced the consistency gap by approximately 50%. As the number of questions per case increased, the consistency gap tended to increase across all evaluated LLMs.</p><p><strong>Conclusions: </strong>Current LLMs exhibit high accuracy in answering individual questions, but their ability to correctly answer all questions within a complete clinical case is substantially lower. This study design evaluated independent question answering, with responses aggregated at the case level. The performance depends heavily on model scale and case complexity. LLMs should be used as support tools rather than independent decision-makers in medical education and clinical practice.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e95342"},"PeriodicalIF":13.9,"publicationDate":"2026-09-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13643263/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"149026190","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}