JMIR Medical Education最新文献

{"title":"Profiling Human-AI Regulatory Support in Technology-Enhanced Interprofessional Education Among Health Professions Students: Person-Centered Exploratory Study.","authors":"Fraide A Ganotice, John Ian Wilzon T Dizon, Xiaoai Shen","doi":"10.2196/95772","DOIUrl":"10.2196/95772","url":null,"abstract":"<p><strong>Background: </strong>Technology-enhanced health care interprofessional education (IPE) places high demands on students' self-regulated learning (SRL) and their ability to work productively with others to prepare them for collaborative practice in health care settings. Yet, little is known about how health professions students perceive and combine their own SRL with coregulation from human and AI-based support in such environments.</p><p><strong>Objective: </strong>This exploratory and observational study adopted a person-centered approach to identify regulatory profiles based on self-reported SRL and perceived coregulation with near-peer teachers (NPTs) and generative AI (GenAI), and to examine how these profiles were associated with interprofessional learning outcomes.</p><p><strong>Methods: </strong>Health professions students (N=136) enrolled in a technology-enhanced IPE completed an SRL questionnaire at the beginning of the program. They rated coregulation from NPTs at midprogram and coregulation from GenAI at the end, together with communication, collaboration, and learning satisfaction. A 2-stage exploratory clustering approach using SRL and coregulation scores as indicators was used to identify distinct regulatory profiles. Profile differences in interprofessional communication, collaboration, and learning satisfaction were examined using independent samples 2-tailed <i>t</i> tests.</p><p><strong>Results: </strong>A 2-profile solution provided the best fit to the data. The positive human-NPT-GenAI regulation profile (51/136, 37.5%) was characterized by moderately high SRL, high coregulation with NPTs, and above-average coregulation with GenAI. The negative human-NPT-GenAI regulation profile (85/136, 62.5%) showed the opposite pattern, with moderately low SRL, low coregulation with NPTs, and below-average coregulation with GenAI. Students in the positive profile reported significantly higher interprofessional communication (<i>P</i>=.005) and collaboration (<i>P</i>=.006) and greater learning satisfaction (<i>P</i><.001) than those in the negative profile, with medium to medium-large effect sizes.</p><p><strong>Conclusions: </strong>These findings provide person-centered evidence that self-regulation and coregulation from human and AI agents form distinct perceived regulatory configurations in technology-enhanced IPE and that a richer \"regulatory ecology,\" combining stronger SRL with greater perceived coregulation from near-peer and GenAI support, was associated with more favorable interprofessional outcomes. The study highlights the importance of deliberately designing near-peer teaching and GenAI-supported activities as complementary regulatory scaffolds in health professions education.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e95772"},"PeriodicalIF":13.9,"publicationDate":"2026-09-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13561188/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148931056","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"A Dual Hospital-University Specialist Training Reform to Address Workforce Shortage and Geographic Maldistribution in Indonesia: Integrative Qualitative Policy Process Review.","authors":"Asnawi Abdullah, Yuli Farianti, Lupi Trilaksono, Arianti Anaya, Anna Kurniati, Oos Fatimah Rosyati, Etik Retno Wiyati, Mimi Sumiarsih, Christa Gumanti Manik, Nyiayu H A Sonia, Farizal Rizky Muharram, Kevin Tadeus Simanjuntak, Sydney Tjandra, Akemat Akemat, Else Mutiara Sihotang, Leni Kuswandari, Siti Yunianti, Fialisa Asriwhardani","doi":"10.2196/90270","DOIUrl":"10.2196/90270","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Indonesia faces 3 interlocking medical workforce crises: an absolute specialist deficit projected to reach 70,000 by 2032 (national density of 0.18 per 1000 population vs the Ministry of National Development Planning [Bappenas] target of 0.28), severe maldistribution (with nearly 59% of specialists concentrated in Java), and a structural anomaly in which residents pay tuition while performing essential clinical work. The 2023 Health Law (Law 17/2023) authorized a transformative reform: a hospital-based residency pathway (Rumah Sakit Pendidikan Penyelenggara Utama [primary teaching hospital; RSPPU]) operating in parallel with the long-established university-based system.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to examine the rationale, policy design, and early implementation of Indonesia's dual hospital-university specialist medical education reform, interpreted through an 8-step change management framework developed by Kotter, and to identify transferable lessons for low- and middle-income countries.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;We conducted an integrative qualitative policy process review combining two evidence streams: (1) systematic documentary analysis of 19 source documents (17 primary legal, regulatory, and policy instruments plus 2 interministerial joint monitoring site-visit reports) and (2) engagement of 34 key informants through semistructured interviews and focus group discussions (45-120 min), comprising policymakers, collegium representatives, hospital leaders, and residents across all 6 pilot sites, recruited purposively until thematic saturation. Interview and focus group data were analyzed using a hybrid deductive-inductive thematic approach with an 8-step change management framework developed by Kotter as an a priori coding frame, and member checking was completed with 7 of the 34 informants.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;The reform designated 6 top-tier national referral hospitals as RSPPUs and enrolled 52 residents from 412 applicants (an acceptance rate of 12.6%) across 6 high-need specialties (ophthalmology, cardiology, pediatrics, orthopedics, neurology, and oncology). All 6 pilot sites established functional education units and designated institutional officials, adopted dual accreditation, and operationalized an integrated e-logbook for competency tracking, real-time monitoring of 80-hour duty limits, and anonymous bullying reporting. An interministerial joint monitoring team visited all 6 sites and scored each site as satisfactory or better across governance, curriculum, faculty, infrastructure, and learner support. Four cross-cutting themes emerged: financial-barrier removal, dual-governance pragmatism, accreditation strain, and equity-anchored deployment. Persistent tensions include variable educator compensation across hospitals; however, a standardized national framework remains under development. Mapping to the framework developed by Kotter demonstrated strong evidence ","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e90270"},"PeriodicalIF":13.9,"publicationDate":"2026-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13557448/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148921158","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"AI-Assisted Angoff Standard Setting for Multiple-Choice Examinations in Medical Education: Comparative Study.","authors":"Salah Eldin Kassab, Mariam Shadan, Arina Ziganshina, Shifan Khanday, Hossam Hamdy","doi":"10.2196/98766","DOIUrl":"10.2196/98766","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Standard setting is essential for a defensible assessment in medical education. The modified Angoff method requires several expert judges, and determining the minimally competent candidate is cognitively challenging. However, empirical evidence on the role of AI in standard setting is unclear.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to examine the role of large language models (LLMs) in the modified Angoff standard setting method for multiple-choice questions in a basic medical science examination compared with faculty judges.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;This study was conducted in year 2 of the MD program in the United Arab Emirates. Ten faculty judges and 5 LLMs (GPT-5.2, Grok, DeepSeek, MedGemma, and Claude 4.5 Sonnet) determined the Angoff cutoff scores for a summative examination (120 multiple-choice questions). Standardized prompts were used for the LLMs to mimic the same information provided for faculty judges. Generalizability (&lt;i&gt;G&lt;/i&gt;) theory analyses were performed using a fully crossed item × rater design to estimate variance components, &lt;i&gt;G&lt;/i&gt; and Φ coefficients, decision study, and root mean squared error (RMSE) of the Angoff cutoff scores.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;Faculty-generated Angoff estimates (mean 69.13, SD 10.92) were comparable to LLM-generated estimates (mean 68.94, SD 12.39). A 2-tailed paired-sample &lt;i&gt;t&lt;/i&gt; test revealed no statistically significant difference between the 2 groups (95% CI -2.21 to 2.91; &lt;i&gt;t&lt;/i&gt;&lt;sub&gt;119&lt;/sub&gt;=0.27; &lt;i&gt;P&lt;/i&gt;=.79). Generalizability theory analysis demonstrated moderate reliability for the faculty panel (&lt;i&gt;G&lt;/i&gt; coefficient=0.738; Φ coefficient=0.692). Despite comprising only 5 LLMs, the LLM panel demonstrated higher reliability (&lt;i&gt;G&lt;/i&gt; coefficient=0.823; Φ coefficient=0.815), lower RMSE (1.28 vs 2.33), higher item-related variance (46.85% vs 18.36%), and lower rater-related variance (2.64% vs 16.40%) than the faculty panel. Pass rates were similar using LLM- and faculty-derived cutoff scores (52/73, 71.2%). LLMs differed in their minimally competent candidate conceptualization and approaches to determining item-level percentages of correct responses. Furthermore, the correlations between Angoff estimates and item-related &lt;i&gt;P&lt;/i&gt; values were larger in LLMs than in faculty judges (&lt;i&gt;r&lt;/i&gt;=0.552 vs 0.437).&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;In this single-institution study, the evaluated LLMs generated modified Angoff estimates that were broadly comparable to those of faculty judges. Generalizability theory analyses demonstrated higher &lt;i&gt;G&lt;/i&gt; and Φ coefficients, lower rater-related variance, and lower RMSE for the evaluated LLM outputs under standardized prompting conditions, indicating greater consistency in the generated estimates. Application of the resulting cutoff scores produced pass and fail rates similar to those derived from faculty judges. These findings suggest the future application of LLMs as decision assistance to","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e98766"},"PeriodicalIF":13.9,"publicationDate":"2026-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13557613/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148921255","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"How to Conduct Usability Testing of Medical Education Simulations: Methods Tutorial Using a Telemedicine Simulation for Maternal Care.","authors":"Blake J Lesselroth, Alexander Douglas, Helen Monkman, Romaric Marcilly, Karalane Bellavia, Mataeo Anderson, Jacob Van Buren, Charles Parsons, Sonakshi Mohanty, Juell Homco, Morgan Richards, Karen P Gold","doi":"10.2196/102697","DOIUrl":"10.2196/102697","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Clinical educators regularly use simulation-based education to help learners develop clinical reasoning, procedural skills, and communication strategies. However, differences between simulated and live clinical environments, such as missing case details or workflow interruptions, can reduce realism and divert learners' attention from the intended learning goals. While published design guidelines recommend pilot testing simulations, they do not explain how to systematically identify usability problems. Human factors evaluation methods can address this gap by identifying issues before piloting and providing design insights.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This tutorial describes our usability testing protocol for evaluating educational simulations before piloting with learners. Using a telemedicine maternal care simulation as an example, we show how mixed methods usability testing can identify technical, workflow, and assessment-related issues in simulation design.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;We recruited obstetricians and family practitioners to test our simulation. To measure multiple usability dimensions concurrently, we used several published instruments, including an Agile Task Analysis (ATA), the Single Ease Question (SEQ), an adapted System Usability Scale (SUS), the Satisfaction With Simulation Experience Scale (SSES), and the NASA (National Aeronautics and Space Administration) Task Load Index (NASA-TLX). We also developed a competency assessment rubric to measure telemedicine competencies. We recorded observations on the ATA while testers used a think-aloud protocol. We then debriefed testers and administered the posttest questionnaires. We summarized quantitative data using descriptive statistics and analyzed qualitative data using theory-based deductive coding with published usability heuristics.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;Twelve testers identified 52 usability issues over 2 testing cycles; 15 issues required correction before piloting with learners. Revisions included changes to standardized patient scripts, synthetic patient data, and scoring rubric instructions. Testers assigned a median SEQ score of 6 or higher to 4 of the 6 workflow steps; most struggled with unfamiliar telemedicine tasks or completing a virtual physical examination. SUS scores ranged from 65 to 100. The median SUS score was 77.5, indicating above-average usability. The NASA-TLX subscale scores were highest for mental workload (median 60, IQR 56.3-63.8) compared with other subscales, such as physical or temporal workload (median 15.00, IQR 11.3-18.8 and median 20.00, IQR 11.3-43.8, respectively), suggesting the simulation imposed a high mental demand. The overall SSES score of 43.5 out of 50 suggested faculty testers thought the simulation offered valuable learning opportunities.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;Following usability testing, we successfully piloted the simulation with 23 obstetrics and family medicine re","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e102697"},"PeriodicalIF":13.9,"publicationDate":"2026-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13601883/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148921234","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Effect of Large Language Model-Powered Virtual Standardized Patients on History-Taking Among Undergraduate Medical Students: Propensity-Matched Cohort Study.","authors":"Yuchen He, Chen Chen, Rong Yin, Wuyang Zhang, Haoli Chang, Wei Yang, Fei Li, Xinhua Li, Zhuying Xia, Xiaoyun Xie, Jing Huang, Qiuming Zeng, Guang Yang, Jing Wu","doi":"10.2196/92486","DOIUrl":"10.2196/92486","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Medical history taking (MHT) is a foundational clinical competency for medical students; however, traditional training models using standardized patients face challenges such as resource constraints. Large language model-powered virtual standardized patients (LLM-VSPs) offer a safe, repeatable platform for self-directed practice with AI-automated feedback. Nevertheless, their effectiveness in authentic teaching environments and underlying learning mechanisms require further investigation.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aims to evaluate the impact of an LLM-VSP system as an extracurricular self-practice tool on undergraduate medical students' MHT performance in an authentic educational setting without disrupting routine instruction, and to further explore potential associations among practice behaviors, baseline proficiency, and intervention effects.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;This prospective cohort study enrolled 168 third-year medical students. Based on voluntary participation, students were assigned to an intervention group (n=120, using LLM-VSP) or a control group (n=48, receiving routine instruction). Propensity score matching (PSM) balanced confounding factors, yielding 40 matched pairs. Baseline MHT performance was assessed via virtual patient examination after didactic instruction but before clinical practicum. The primary outcome was end-of-term MHT performance assessed at an Objective Structured Clinical Examination station with real standardized patients. The differences between groups were compared using independent samples t test, with robustness validated through multiple linear regression, sensitivity analyses, and Rosenbaum bounds analyses. Exploratory analyses investigated the association between practice behaviors and scores, and observed benefit differences across baseline levels.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;After PSM, baseline characteristics were balanced (standardized mean difference &lt;0.1). The intervention group exhibited higher total MHT scores than controls (mean 87.71, SD 7.29 vs mean 83.74, SD 8.06; mean difference 3.98, 95% CI 0.55-7.40 points; P=.02; with a medium effect size of Cohen d=0.52). Advantages were observed in content (P=.03) and communication skills (P=.04) subscores. Regression analysis confirmed robust intervention effects (B=3.924, 95% CI 1.90-5.94; P&lt;.001; R&lt;sup&gt;2&lt;/sup&gt;=0.708), with sensitivity analyses supporting reliability. Exploratory analysis suggested that mere practice behavior metrics were not independent predictors of final scores, potentially being constrained by baseline proficiency, implying a \"cognitive threshold\" for effective AI-assisted training. Subgroup analysis indicated a trend of differential benefits: high baseline students appeared to show larger gains (matched sample: mean difference 5.93, 95% CI 2.17-9.70; overall sample: mean difference 7.04, 95% CI 3.84-10.25), whereas improvements in medium and low baseline students were","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e92486"},"PeriodicalIF":13.9,"publicationDate":"2026-09-09","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13601875/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148913439","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Role-Differentiated AI Competencies and Curriculum Implications for Health Professions Education: Qualitative Study.","authors":"Yuyi Park, Yejin Han, Hyeongjo Kim, Solmoe Ahn, Frederick William Kron, Jihyun Lee","doi":"10.2196/97608","DOIUrl":"10.2196/97608","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;As AI fundamentally transforms the healthcare landscape, medical education curricula have struggled to keep pace with these technological shifts. While current research has established a foundation for general AI literacy, less attention has been given to the role-specific competencies required for the diverse functions that healthcare professionals perform in an AI-integrated environment. Although existing tiered and domain-based competency models have clarified general AI competency requirements, they offer limited guidance on how such competencies should be differentiated according to the roles healthcare professionals perform in practice.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to explore how AI-driven changes in healthcare shape competency requirements across professional roles and to develop differentiated competency frameworks and curriculum implications for 3 distinct roles: users, developers, and leaders.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;Using a qualitative research design, we conducted in-depth interviews with 13 subject matter experts across academic medicine and dentistry, clinical practice, and the healthcare AI industry, who were recruited through 4 independent channels. Data were analyzed using reflexive thematic analysis with inductive coding organized around 3 research questions and published competency frameworks serving as sensitizing concepts. Trustworthiness was supported through investigator triangulation, member checking, an audit trail, and attention to researcher reflexivity and positionality.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;We identified a role-differentiated competency framework with a layered structure as follows: (1) Users require machine learning and data literacy to understand where algorithms fail and scrutinize data origins and biases; the ability to select best-fit AI solutions and set appropriate thresholds for human-machine delegation; and a high degree of AI-related professionalism to work responsibly with AI by recognizing its limits, critically appraising outputs, maintaining clinical accountability, preserving patient-centered care, and updating knowledge continuously; (2) Developers add operational competencies, including validating systems across technical performance, clinical relevance, and regulatory compliance; bridging the language gap between clinical needs and technical refinements; and fostering a challenging, problem-solving mindset to address real-world clinical bottlenecks; and (3) Leaders focus on system-level competencies, including macro-level strategic planning, governing medical AI across its full lifecycle-selection, validation, deployment, and monitoring-through policy \"roads\" and governance frameworks, and optimizing systemic resource utilization while orchestrating trust-based collaboration that safeguards care quality. Rather than a strictly cumulative hierarchy, the roles represent analytic distinctions with shared foundational competencies. ","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e97608"},"PeriodicalIF":13.9,"publicationDate":"2026-09-08","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13552830/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148913525","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Multimodal Large Language Models for Dental Chart Image Interpretation: Cross-Sectional Benchmarking Study With Students and Clinicians.","authors":"Ah-Young Cho, Soo-Heang Eo, Mi-Jeong Jeon, Jae-Hoon Kim, Jungjoon Ihm, Deog-Gyu Seo","doi":"10.2196/91809","DOIUrl":"10.2196/91809","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;Entering clinical training, dental students must learn to read tooth-centered electronic dental records, but limited teaching time in patient-centered clinics can leave gaps in chart-reading literacy. Multimodal large language models (LLMs) that process dental record images may offer scalable educational support, yet their performance has not been benchmarked.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study evaluated multimodal ChatGPT models on dental chart image interpretation and compared the best-performing model with dental students and residents.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;A retrospective, cross-sectional benchmark study used deidentified dental chart images from 15 patients at Seoul National University Dental Hospital (2017-2025). Charts containing Korean and English text were captured as sequential screenshots (154 images). For each patient, 24 Korean-language questions (360 total) spanned four categories: Type A, general factual retrieval; Type B, tooth- or procedure-specific retrieval; Type C, interpretation requiring multientry synthesis; and Type D, absent-information questions assessing abstention. Nine multimodal ChatGPT models (available August 2, 2025-August 9, 2025) were evaluated under standardized conditions. Outputs were scored against a gold standard using 7 metrics, with sentence bidirectional encoder representations from transformers (SBERT) similarity prespecified as the primary semantic measure. Human baselines included 2 third-year students and 2 first-year residents. Groups were compared with Kruskal-Wallis tests and Dunn post hoc analyses. Gold-standard reliability was assessed by independent senior-expert review and chance-corrected agreement (Gwet AC1) among clinical reference raters, and the model ranking was confirmed by content-based clinical accuracy analysis.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;Across 360 items, GPT-5 Thinking achieved the highest median SBERT similarity (0.900, IQR 0.525-1.000), followed by GPT-5 Pro (0.861, IQR 0.501-1.000) and OpenAI o3 (0.831, IQR 0.489-1.000), with significant overall group differences (&lt;i&gt;P&lt;/i&gt;&lt;.001). Student 1 did not differ significantly from GPT-5 Thinking across Types A-D (all &lt;i&gt;P&lt;/i&gt;≥.16), whereas Student 2 differed only on Type D (&lt;i&gt;P&lt;/i&gt;=.002; Cliff δ=-0.27). Resident 1 scored higher than GPT-5 Thinking on Types A (&lt;i&gt;P&lt;/i&gt;=.01) and C (&lt;i&gt;P&lt;/i&gt;=.04) but lower on Type D (&lt;i&gt;P&lt;/i&gt;&lt;.001; δ=-0.35), whereas Resident 2 scored higher on Types A (&lt;i&gt;P&lt;/i&gt;=.03), B (&lt;i&gt;P&lt;/i&gt;=.02), and C (&lt;i&gt;P&lt;/i&gt;=.04) and did not differ on Type D (&lt;i&gt;P&lt;/i&gt;=.23). Type C tasks showed compressed SBERT distributions and low exact-match rates, indicating persistent difficulty in synthesis. The clinical reference rater agreement was high (Gwet AC1=0.99), and the content-based clinical-accuracy ranking was closely aligned with the SBERT ranking (Spearman ρ=0.90; &lt;i&gt;P&lt;/i&gt;=.001).&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;Statistically nonsignificant differences were observed ","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e91809"},"PeriodicalIF":13.9,"publicationDate":"2026-09-04","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13544698/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148892051","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"What Platform Scores Miss: Multidimensional Evaluation of AI Teaching Agents in Medical Education.","authors":"Hui Zhang, Lihui Qu, Jianmin Zheng, Yi Xiong, Hongbo Bai, Ruiying Ji, Guohui Liu, Wanling Chen, Zirui Cheng, Youbang Chen, Chun-Tao Yang","doi":"10.2196/96819","DOIUrl":"10.2196/96819","url":null,"abstract":"<p><strong>Background: </strong>Large language model (LLM)-based AI teaching agents are increasingly used in medical education, yet their pedagogical quality is typically judged by platform-generated scores whose scoring criteria are undisclosed and may not reflect the teaching quality of the agent.</p><p><strong>Objective: </strong>This study aimed to develop and validate a multidimensional rubric for evaluating AI teaching agents and to examine the correspondence between platform scores and rubric-based teaching quality.</p><p><strong>Methods: </strong>Eight AI teaching agents covering an endocrinology curriculum were deployed across 4 role-play paradigms (patient, student, expert, and family). Twenty-two fourth-year medical students generated 167 dialogues, which were scored both by the platform and by an independently applied 8-dimension rubric (100 points, covering knowledge accuracy, pedagogical guidance, knowledge coverage, role-play quality, difficulty calibration, medical safety, student engagement, and feedback quality). Each dialogue was scored 4 times by a primary evaluator (Claude Opus 4.8; mean within-model SD 0.36), with 2 additional LLMs as robustness checks; 40 dialogues spanning all agents were rescored by a medical-education expert for validation.</p><p><strong>Results: </strong>Platform and rubric rankings diverged for most agents: the agent ranked third by the platform ranked last on rubric-based quality, and the platform's fourth-ranked agent ranked first. Agents differed most on knowledge-related dimensions (knowledge coverage coefficient of variation=27.3%) and least on role-play quality (coefficient of variation=5.7%), while difficulty calibration was a shared weakness. In a case-level observation, one agent revised specifically to strengthen empathy attained high role-play quality yet the lowest knowledge coverage of all agents. AI scores agreed with expert ratings at the total-score level (intraclass correlation coefficient=0.51) and on cognitive-process dimensions, but agreement was low for the more subjective dimensions. Student gender showed no detectable effect, though this analysis was underpowered.</p><p><strong>Conclusions: </strong>In this exploratory study, platform-generated scores reflected a construct different from agent teaching quality and should be used as a complement rather than as the sole quality indicator. The 8-dimension rubric provides a transparent, standardized alternative that reveals differences missed by platform scores, including a lack of association between empathy and knowledge coverage that warrants attention in future agent design.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e96819"},"PeriodicalIF":13.9,"publicationDate":"2026-09-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13540620/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148888868","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Scaffolded AI-Supported Problem-Based Learning for Medical Interns: Exploratory Retrospectively Registered Randomized Controlled Evaluation With a Voluntary Feasibility Follow-Up.","authors":"Zhengqi Zhao, Baijing Wu, Shuyuan Tian, Haigen Hu, Pengyi Hao","doi":"10.2196/95039","DOIUrl":"10.2196/95039","url":null,"abstract":"&lt;p&gt;&lt;strong&gt;Background: &lt;/strong&gt;High-quality problem-based learning (PBL) during internship is resource-intensive and difficult to scale without consistent facilitation. Although generative AI is increasingly used in health professions education, many applications remain on-demand answer tools that may not reproduce core PBL processes.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Objective: &lt;/strong&gt;This study aimed to develop and evaluate the Multiagent PBL Environment for Clinical Reasoning (MAPLE-CR), an AI-supported environment that uses generative AI as a process-oriented scaffold rather than an answer-delivery aid. The system supports cognitive mechanisms through reasoning prompts, social-interactional mechanisms through simulated tutor and peer roles, and regulatory mechanisms through structured workflows and feedback loops. We separately evaluated implementation and repeated-use feasibility, including learner experience, short-term outcomes versus self-study, and preliminary framework-aligned process evidence.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Methods: &lt;/strong&gt;We conducted a 2-stage study. In an exploratory randomized evaluation (N=52; intervention: n=26; control: n=26), interns completed parallel pretests and posttests around a standardized case. The intervention group engaged in asynchronous, scaffolded PBL in MAPLE-CR, whereas controls completed case-matched self-study using materials derived from the same case and learning objectives. We summarized transcript-derived process dimensions and postsession learner experience in the intervention group. In a voluntary follow-up, 18 participants completed 1 MAPLE-CR case per week for 4 additional weeks to examine repeated use feasibility and patterns across novel cases.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Results: &lt;/strong&gt;Baseline pretest scores were comparable between groups (&lt;i&gt;P&lt;/i&gt;=.56). The intervention group achieved higher posttest scores than controls (Hodges-Lehmann median difference 8.25 points, 95% CI 6.60-11.60; Cliff δ=0.506, 95% CI 0.21-0.76; &lt;i&gt;P&lt;/i&gt;=.001) and greater score gains (median difference 6.60 points, 95% CI 1.60-14.95; &lt;i&gt;P&lt;/i&gt;=.01). On a 0 to 5 scale, mean scores were highest for knowledge accuracy (4.4) and clinical reasoning (CR, 3.7), whereas active participation averaged 2.1, and interaction-oriented dimensions were more variable. Means for 12 positively worded questionnaire items ranged from 4.19 to 4.62, supporting high satisfaction, involvement, perceived support, and self-efficacy; open-ended responses contextualized perceived strengths and improvement needs. In the voluntary follow-up (n=18), weekly cross-case use was feasible; median within-session gains ranged from 13.4 to 20.0 points across 5 attempts, with marked increases in CR and aggregate scores from attempts 1 to 2 but statistically uncertain later trajectories.&lt;/p&gt;&lt;p&gt;&lt;strong&gt;Conclusions: &lt;/strong&gt;MAPLE-CR was feasible and showed high learner acceptability. Its use was associated with larger short-term CR knowledge gains than case-matched self-study, while transcript ","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e95039"},"PeriodicalIF":13.9,"publicationDate":"2026-09-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13541230/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148888698","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Estimating the Prevalence of Generative AI Use in Medical School Application Essays: Cross-Sectional Study.","authors":"Nicholas C Spies, Valerie S Ratts, Ian S Hagemann","doi":"10.2196/96673","DOIUrl":"10.2196/96673","url":null,"abstract":"<p><strong>Background: </strong>Generative AI tools became widely available to the public in November 2022. The extent to which these tools have been used by medical school applicants during the admissions process is unknown.</p><p><strong>Objective: </strong>We aimed to estimate the extent of generative AI use among cohorts of applicants spanning the rollout of these tools.</p><p><strong>Methods: </strong>We retrospectively analyzed 6000 essays from 2364 applicants submitted to a US medical school in 2021 to 2022 (baseline, before the wide availability of AI) and 2023 to 2024 (test year) to estimate the prevalence of AI use and its relation to other application data. We used GPTZero, a commercially available detection tool, to generate a metric (P_human) reflecting the predicted probability that each essay was completely human generated, ranging from 0 (the essay appears to be entirely AI generated) to 1 (the essay appears to be entirely human generated).</p><p><strong>Results: </strong>Fully human-generated negative controls demonstrated a median P_human of 0.93 (range 0.89-0.97), while fully AI-generated positive controls demonstrated a median P_human of 0.01 (range 0.00-0.01). The \"Personal Comments\" essays submitted in the 2023 to 2024 application cycle had a median P_human of 0.77 (95% CI 0.76-0.78) compared with 0.83 (95% CI 0.82-0.85) during the 2021 to 2022 cycle. Approximately 12.3% and 2.7% of essays were evaluated as having P_human <0.5 in the test and baseline years, respectively. Essays submitted as part of the secondary application demonstrated lower P_human values than those of the American Medical College Application Service (AMCAS) \"Personal Comments\" essays. In applicant-clustered, multivariable generalized estimating equation analyses, supplementary essay type and younger age were significantly associated with lower P_human. Application completion date, self-reported gender, program type (MD vs MD-PhD), grade point average (GPA), Medical College Admission Test (MCAT) score, socioeconomic status, and undergraduate major were not significant predictors after false discovery rate correction. P_human was not predictive of interview invitation or acceptance in adjusted applicant-level logistic regression analyses.</p><p><strong>Conclusions: </strong>An AI detection algorithm identified signs of increased use of generative AI in 2023 to 2024 medical school admission applications compared to those in the 2021 to 2022 baseline period, before AI was widely available. AI use did not appear to confer an admissions advantage. Although these results provide information about the applicant pool as a whole, AI detection is imperfect. We do not recommend deploying AI detection for individual applications in live admissions cycles.</p>","PeriodicalId":36236,"journal":{"name":"JMIR Medical Education","volume":"12 ","pages":"e96673"},"PeriodicalIF":13.9,"publicationDate":"2026-09-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13586704/pdf/","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"148888687","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"OA","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}