Using randomization to compare AI and expert-generated formative assessment questions in medical education.

IF 4.4 2区 医学 Q1 EDUCATION & EDUCATIONAL RESEARCH
Medical Education Online Pub Date : 2026-12-31 Epub Date: 2026-05-09 DOI:10.1080/10872981.2026.2671586
Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed
{"title":"Using randomization to compare AI and expert-generated formative assessment questions in medical education.","authors":"Akshat Kumar, Katie Stinson, Litao Wang, Elizabeth Nugent, Yongsu Lee, Mark Hormann, LaTanya Jones Love, Toufeeq Ahmed Syed","doi":"10.1080/10872981.2026.2671586","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>AI-generated content is being used across the education spectrum and is beneficial for creating complex multiple-choice questions, such as those used in medical education. However, evaluating AI-generated content is challenging, and existing testing and evaluation methods are falling short. This study uses randomization to compare medical students' performance on and subjective evaluation of AI versus expert-generated questions. We hypothesized that there would be no difference in student performance or subjective evaluation of AI vs expert-generated questions.</p><p><strong>Methods: </strong>We designed a single-center, randomized study where medical student participants received either one AI- or expert-generated question per day over four weeks.</p><p><strong>Results: </strong>Our study showed that participants had similar perceptions of AI versus expert-generated questions (<i>p</i> = 0.18), with no significant difference between the distributions of the proportion of correct responses. The cumulative proportion of questions answered correctly over 28 days was consistent across the two question sets. However, participants rated 53% of AI-generated questions as very easy or easy compared with only 31% of the expert-generated questions.</p><p><strong>Discussion: </strong>Randomization was crucial to show that AI-generated questions were nearly imperceptible from expert-generated questions, demonstrating the need for additional evaluation methods to compare AI- and expert-generated medical education content.</p>","PeriodicalId":47656,"journal":{"name":"Medical Education Online","volume":"31 1","pages":"2671586"},"PeriodicalIF":4.4000,"publicationDate":"2026-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13159578/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Medical Education Online","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1080/10872981.2026.2671586","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2026/5/9 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"EDUCATION & EDUCATIONAL RESEARCH","Score":null,"Total":0}
引用次数: 0

Abstract

Background: AI-generated content is being used across the education spectrum and is beneficial for creating complex multiple-choice questions, such as those used in medical education. However, evaluating AI-generated content is challenging, and existing testing and evaluation methods are falling short. This study uses randomization to compare medical students' performance on and subjective evaluation of AI versus expert-generated questions. We hypothesized that there would be no difference in student performance or subjective evaluation of AI vs expert-generated questions.

Methods: We designed a single-center, randomized study where medical student participants received either one AI- or expert-generated question per day over four weeks.

Results: Our study showed that participants had similar perceptions of AI versus expert-generated questions (p = 0.18), with no significant difference between the distributions of the proportion of correct responses. The cumulative proportion of questions answered correctly over 28 days was consistent across the two question sets. However, participants rated 53% of AI-generated questions as very easy or easy compared with only 31% of the expert-generated questions.

Discussion: Randomization was crucial to show that AI-generated questions were nearly imperceptible from expert-generated questions, demonstrating the need for additional evaluation methods to compare AI- and expert-generated medical education content.

采用随机化方法比较医学教育中人工智能和专家生成的形成性评估问题。
背景:人工智能生成的内容正被广泛用于教育领域,有利于创建复杂的选择题,例如医学教育中使用的选择题。然而,评估人工智能生成的内容具有挑战性,现有的测试和评估方法存在不足。本研究采用随机方法比较医学生在人工智能和专家生成问题上的表现和主观评价。我们假设学生的表现或对人工智能和专家生成的问题的主观评价没有差异。方法:我们设计了一项单中心随机研究,医学生参与者在四周内每天收到一个人工智能或专家生成的问题。结果:我们的研究表明,参与者对人工智能和专家生成的问题有相似的看法(p = 0.18),正确回答比例的分布之间没有显著差异。在28天内,两个问题集答对问题的累积比例是一致的。然而,参与者将53%的人工智能生成的问题评为非常简单或简单,而专家生成的问题只有31%。讨论:随机化对于表明人工智能生成的问题与专家生成的问题几乎难以察觉至关重要,这表明需要额外的评估方法来比较人工智能和专家生成的医学教育内容。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
Medical Education Online
Medical Education Online EDUCATION & EDUCATIONAL RESEARCH-
CiteScore
6.00
自引率
2.20%
发文量
97
审稿时长
8 weeks
期刊介绍: Medical Education Online is an open access journal of health care education, publishing peer-reviewed research, perspectives, reviews, and early documentation of new ideas and trends. Medical Education Online aims to disseminate information on the education and training of physicians and other health care professionals. Manuscripts may address any aspect of health care education and training, including, but not limited to: -Basic science education -Clinical science education -Residency education -Learning theory -Problem-based learning (PBL) -Curriculum development -Research design and statistics -Measurement and evaluation -Faculty development -Informatics/web
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书