Can Large Language Models Replicate Systematic Review Outcome Classifications in Medical Education? A Pilot Study Using Kirkpatrick Levels.

IF 1.8 Q2 EDUCATION, SCIENTIFIC DISCIPLINES

Medical Science Educator Pub Date : 2026-01-16 eCollection Date: 2026-02-01 DOI:10.1007/s40670-026-02639-1

Giuliano Romano, Emilio Romano, Michelle Rau

引用次数: 0

Abstract

Systematic reviews in medical education often classify outcomes using the Kirkpatrick framework, but manual coding is time-consuming and subjective. We conducted a proof-of-concept study testing ChatGPT (GPT-5, August 2025 release) on 32 full-text articles from a published systematic review of sepsis education. Agreement with human-coded outcomes was modest: 50% percent agreement, unweighted κ = 0.170 (95% CI 0.000-0.458), weighted κ = 0.351 (95% CI 0.074-0.629). Most disagreements were between adjacent levels.

Supplementary information: The online version contains supplementary material available at 10.1007/s40670-026-02639-1.

查看原文本刊更多论文

大型语言模型能否复制医学教育系统评价结果分类？使用柯克帕特里克水平的试点研究。

医学教育中的系统评价通常使用Kirkpatrick框架对结果进行分类，但手工编码既耗时又主观。我们对ChatGPT （GPT-5, 2025年8月发布）进行了一项概念验证研究，测试了32篇来自已发表的败血症教育系统综述的全文文章。与人类编码结果的一致性不高：50%的一致性，未加权κ = 0.170 (95% CI 0.000-0.458)，加权κ = 0.351 （95% CI 0.074-0.629）。大多数分歧发生在相邻级别之间。补充资料：在线版本提供补充资料，网址为10.1007/s40670-026-02639-1。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Medical Science Educator Social Sciences-Education

CiteScore

2.90

自引率

11.80%

发文量

202

期刊介绍： Medical Science Educator is the successor of the journal JIAMSE. It is the peer-reviewed publication of the International Association of Medical Science Educators (IAMSE). The Journal offers all who teach in healthcare the most current information to succeed in their task by publishing scholarly activities, opinions, and resources in medical science education. Published articles focus on teaching the sciences fundamental to modern medicine and health, and include basic science education, clinical teaching, and the use of modern education technologies. The Journal provides the readership a better understanding of teaching and learning techniques in order to advance medical science education.