Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed
{"title":"Using randomization to compare AI and expert-generated formative assessment questions in medical education.","authors":"Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed","doi":"10.1080/10872981.2026.2671586","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>AI-generated content is being used across the education spectrum and is beneficial for creating complex multiple-choice questions, such as those used in medical education. However, evaluating AI-generated content is challenging, and existing testing and evaluation methods are falling short. This study uses randomization to compare medical students' performance on and subjective evaluation of AI versus expert-generated questions. We hypothesized that there would be no difference in student performance or subjective evaluation of AI vs expert-generated questions.</p><p><strong>Methods: </strong>We designed a single-center, randomized study where medical student participants received either one AI- or expert-generated question per day over four weeks.</p><p><strong>Results: </strong>Our study showed that participants had similar perceptions of AI versus expert-generated questions (<i>p</i> = 0.18), with no significant difference between the distributions of the proportion of correct responses. The cumulative proportion of questions answered correctly over 28 days was consistent across the two question sets. However, participants rated 53% of AI-generated questions as very easy or easy compared with only 31% of the expert-generated questions.</p><p><strong>Discussion: </strong>Randomization was crucial to show that AI-generated questions were nearly imperceptible from expert-generated questions, demonstrating the need for additional evaluation methods to compare AI- and expert-generated medical education content.</p>","PeriodicalId":47656,"journal":{"name":"Medical Education Online","volume":"31 1","pages":"2671586"},"PeriodicalIF":4.4000,"publicationDate":"2026-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13159578/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Education Online","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1080/10872981.2026.2671586","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/5/9 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0
Abstract
Background: AI-generated content is being used across the education spectrum and is beneficial for creating complex multiple-choice questions, such as those used in medical education. However, evaluating AI-generated content is challenging, and existing testing and evaluation methods are falling short. This study uses randomization to compare medical students' performance on and subjective evaluation of AI versus expert-generated questions. We hypothesized that there would be no difference in student performance or subjective evaluation of AI vs expert-generated questions.
Methods: We designed a single-center, randomized study where medical student participants received either one AI- or expert-generated question per day over four weeks.
Results: Our study showed that participants had similar perceptions of AI versus expert-generated questions (p = 0.18), with no significant difference between the distributions of the proportion of correct responses. The cumulative proportion of questions answered correctly over 28 days was consistent across the two question sets. However, participants rated 53% of AI-generated questions as very easy or easy compared with only 31% of the expert-generated questions.
Discussion: Randomization was crucial to show that AI-generated questions were nearly imperceptible from expert-generated questions, demonstrating the need for additional evaluation methods to compare AI- and expert-generated medical education content.
期刊介绍:
Medical Education Online is an open access journal of health care education, publishing peer-reviewed research, perspectives, reviews, and early documentation of new ideas and trends.
Medical Education Online aims to disseminate information on the education and training of physicians and other health care professionals. Manuscripts may address any aspect of health care education and training, including, but not limited to:
-Basic science education
-Clinical science education
-Residency education
-Learning theory
-Problem-based learning (PBL)
-Curriculum development
-Research design and statistics
-Measurement and evaluation
-Faculty development
-Informatics/web