Enhancing search strategies for systematic reviews on drug Harms: An evaluation of the utility of ChatGPT in error detection and keyword generation

IF 7 2区医学 Q1 BIOLOGY

Computers in biology and medicine Pub Date : 2025-05-28 DOI:10.1016/j.compbiomed.2025.110464

Victor Gitman, Colleen Maxwell, John-Michael Gamble

{"title":"Enhancing search strategies for systematic reviews on drug Harms: An evaluation of the utility of ChatGPT in error detection and keyword generation","authors":"Victor Gitman, Colleen Maxwell, John-Michael Gamble","doi":"10.1016/j.compbiomed.2025.110464","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>Developing search strategies for synthesizing evidence on drug harms requires specialized expertise and knowledge. The aim of this study was to evaluate ChatGPT's ability to enhance search strategies for systematic reviews of drug harms by identifying missing and generating omitted keywords.</div></div><div><h3>Materials and methods</h3><div>A literature search in PubMed identified systematic reviews of drug harms from 10 high-impact journals between 1-Nov-2013 to 27-Nov-2023. Sixteen search strategies used in these reviews were selected each with a single error of omission introduced. ChatGPT's (GPT-4) performance was evaluated based on error detection, similarity between the extracted and generated search strategies via strict and semantic keyword matching, and proportion of omitted keywords generated.</div></div><div><h3>Results</h3><div>ChatGPT identified the introduced errors in all search strategies. Under strict matching, the mean Jaccard's similarity measure was 0.17 (range: 0.00–0.52) and with semantic matching this increased to 0.23 (range: 0.00–0.53). Similarly, the mean proportion of keywords recreated by ChatGPT was 49 % using strict matching increasing to 71 % with semantic matching.</div></div><div><h3>Discussion and conclusion</h3><div>ChatGPT effectively detected errors and generated relevant keywords, showing potential as a tool for evidence retrieval on drug harms.</div></div>","PeriodicalId":10578,"journal":{"name":"Computers in biology and medicine","volume":"193 ","pages":"Article 110464"},"PeriodicalIF":7.0000,"publicationDate":"2025-05-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Computers in biology and medicine","FirstCategoryId":"5","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0010482525008157","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"BIOLOGY","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

Developing search strategies for synthesizing evidence on drug harms requires specialized expertise and knowledge. The aim of this study was to evaluate ChatGPT's ability to enhance search strategies for systematic reviews of drug harms by identifying missing and generating omitted keywords.

Materials and methods

A literature search in PubMed identified systematic reviews of drug harms from 10 high-impact journals between 1-Nov-2013 to 27-Nov-2023. Sixteen search strategies used in these reviews were selected each with a single error of omission introduced. ChatGPT's (GPT-4) performance was evaluated based on error detection, similarity between the extracted and generated search strategies via strict and semantic keyword matching, and proportion of omitted keywords generated.

Results

ChatGPT identified the introduced errors in all search strategies. Under strict matching, the mean Jaccard's similarity measure was 0.17 (range: 0.00–0.52) and with semantic matching this increased to 0.23 (range: 0.00–0.53). Similarly, the mean proportion of keywords recreated by ChatGPT was 49 % using strict matching increasing to 71 % with semantic matching.

Discussion and conclusion

ChatGPT effectively detected errors and generated relevant keywords, showing potential as a tool for evidence retrieval on drug harms.

查看原文本刊更多论文

加强药物危害系统评价的搜索策略：ChatGPT在错误检测和关键字生成中的效用评估

目的制定药物危害综合证据的检索策略需要专门的专业知识。本研究的目的是评估ChatGPT通过识别缺失和生成遗漏关键词来增强药物危害系统评价的搜索策略的能力。在PubMed进行文献检索，确定了2013年11月1日至2023年11月27日期间10份高影响力期刊对药物危害的系统综述。在这些评论中选择了16种搜索策略，每种策略都引入了一个遗漏错误。ChatGPT （GPT-4）的性能评估基于错误检测、通过严格和语义关键字匹配提取的搜索策略与生成的搜索策略之间的相似度以及生成的遗漏关键字的比例。结果schatgpt识别了所有搜索策略中引入的错误。在严格匹配条件下，Jaccard的相似度均值为0.17（范围：0.00-0.52），在语义匹配条件下，Jaccard的相似度均值增加到0.23（范围：0.00-0.53）。同样，ChatGPT使用严格匹配重新创建的关键词的平均比例为49%，使用语义匹配则增加到71%。讨论与结论chatgpt能够有效地发现错误并生成相关关键词，具有作为药物危害证据检索工具的潜力。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Computers in biology and medicine 工程技术-工程：生物医学

CiteScore

11.70

自引率

10.40%

发文量

1086

审稿时长

74 days

期刊介绍： Computers in Biology and Medicine is an international forum for sharing groundbreaking advancements in the use of computers in bioscience and medicine. This journal serves as a medium for communicating essential research, instruction, ideas, and information regarding the rapidly evolving field of computer applications in these domains. By encouraging the exchange of knowledge, we aim to facilitate progress and innovation in the utilization of computers in biology and medicine.