Accuracy of large language models in data extraction from randomized controlled trials in sleep medicine: A proof-of-concept study

IF 11.6 1区医学 Q1 CLINICAL NEUROLOGY

Sleep Medicine Reviews Pub Date : 2025-12-01 Epub Date: 2025-10-30 DOI:10.1016/j.smrv.2025.102192

Zhen Peng , Xingwei Wu , Zongshi Qin , Suhail A. Doi , Luis Furuya-Kanamori , Yi Hong , Lifeng Lin , Haitao Chu , Chang Xu , Mingkai Liu

{"title":"Accuracy of large language models in data extraction from randomized controlled trials in sleep medicine: A proof-of-concept study","authors":"Zhen Peng , Xingwei Wu , Zongshi Qin , Suhail A. Doi , Luis Furuya-Kanamori , Yi Hong , Lifeng Lin , Haitao Chu , Chang Xu , Mingkai Liu","doi":"10.1016/j.smrv.2025.102192","DOIUrl":null,"url":null,"abstract":"<div><div>This proof-of-concept study examined the performance of two prominent large language model <strong>(</strong>LLM)-based AI tools, ChatGPT 4o and Claude 3.5, in extracting data for four specific tasks: group size, event count, mean value, and standard deviation. Utilizing an established database that analyzed data extraction errors in systematic reviews on sleep medicine, we tested the ability of both AI tools to extract data from 648 randomized controlled trials (RCTs) using single- and multiple-sentences prompting approaches. The accuracy of the extracted data was compared to error-corrected metadata, with an overall accuracy reaching up to 71.5 % (95 % CI: 69.3 %, 73.7 %) for Claude and 69.1 % (95 % CI: 66.8 %, 71.3 %) for ChatGPT. Claude demonstrated superior performance over ChatGPT across all tasks, with the largest accuracy difference of up to 12.7 % (OR = 1.70, 95 % CI: 1.38, 2.10). The single-sentence prompt led to lower accuracy compared to the multiple-sentences prompts, with the largest percentage difference being −11.0 % (OR = 0.64, 95 % CI: 0.52, 0.78). Both AI tools achieved strong performance in extracting group size data. These findings underscore the potential of AI tools like Claude, especially when combined with effective prompting strategies like multiple-sentences prompts, to assist data extraction in sleep medicine research.</div></div>","PeriodicalId":49513,"journal":{"name":"Sleep Medicine Reviews","volume":"84 ","pages":"Article 102192"},"PeriodicalIF":11.6000,"publicationDate":"2025-12-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Sleep Medicine Reviews","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S1087079225001455","RegionNum":1,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/10/30 0:00:00","PubModel":"Epub","JCR":"Q1","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

This proof-of-concept study examined the performance of two prominent large language model (LLM)-based AI tools, ChatGPT 4o and Claude 3.5, in extracting data for four specific tasks: group size, event count, mean value, and standard deviation. Utilizing an established database that analyzed data extraction errors in systematic reviews on sleep medicine, we tested the ability of both AI tools to extract data from 648 randomized controlled trials (RCTs) using single- and multiple-sentences prompting approaches. The accuracy of the extracted data was compared to error-corrected metadata, with an overall accuracy reaching up to 71.5 % (95 % CI: 69.3 %, 73.7 %) for Claude and 69.1 % (95 % CI: 66.8 %, 71.3 %) for ChatGPT. Claude demonstrated superior performance over ChatGPT across all tasks, with the largest accuracy difference of up to 12.7 % (OR = 1.70, 95 % CI: 1.38, 2.10). The single-sentence prompt led to lower accuracy compared to the multiple-sentences prompts, with the largest percentage difference being −11.0 % (OR = 0.64, 95 % CI: 0.52, 0.78). Both AI tools achieved strong performance in extracting group size data. These findings underscore the potential of AI tools like Claude, especially when combined with effective prompting strategies like multiple-sentences prompts, to assist data extraction in sleep medicine research.

查看原文本刊更多论文

睡眠医学随机对照试验数据提取中大语言模型的准确性：一项概念验证研究。

这项概念验证研究检查了两个著名的基于大型语言模型（LLM）的人工智能工具ChatGPT 40和Claude 3.5在为四个特定任务提取数据方面的性能：群体规模、事件计数、平均值和标准差。利用已建立的数据库分析睡眠医学系统评价中的数据提取错误，我们测试了这两种人工智能工具使用单句和多句提示方法从648项随机对照试验（rct）中提取数据的能力。将提取数据的准确性与纠错元数据进行比较，Claude的总体准确性达到71.5% (95% CI: 69.3%, 73.7%)， ChatGPT的总体准确性达到69.1% （95% CI: 66.8%, 71.3%）。Claude在所有任务中都表现出优于ChatGPT的性能，最大的准确率差异高达12.7% （OR = 1.70, 95% CI: 1.38, 2.10）。与多句提示相比，单句提示的准确率较低，最大的百分比差异为- 11.0% （OR = 0.64, 95% CI: 0.52, 0.78）。这两种人工智能工具在提取群体规模数据方面都取得了很好的表现。这些发现强调了像Claude这样的人工智能工具的潜力，特别是当与多句提示等有效提示策略相结合时，有助于睡眠医学研究中的数据提取。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Sleep Medicine Reviews 医学-临床神经学

CiteScore

20.10

自引率

3.80%

发文量

107

期刊介绍： Sleep Medicine Reviews offers global coverage of sleep disorders, exploring their origins, diagnosis, treatment, and implications for related conditions at both individual and public health levels. Articles comprehensively review clinical information from peer-reviewed journals across various disciplines in sleep medicine, encompassing pulmonology, psychiatry, psychology, physiology, otolaryngology, pediatrics, geriatrics, cardiology, dentistry, nursing, neurology, and general medicine. The journal features narrative reviews, systematic reviews, and editorials addressing areas of controversy, debate, and future research within the field.