Will ChatGPT-4 improve the quality of medical abstracts?

IF 1.8 4区医学 Q2 PEDIATRICS

Paediatrics & child health Pub Date : 2024-09-12 eCollection Date: 2025-06-01 DOI:10.1093/pch/pxae062

Jocelyn Gravel, Chloé Dion, Mandana Fadaei Kermani, Sarah Mousseau, Esli Osmanlliu

{"title":"Will ChatGPT-4 improve the quality of medical abstracts?","authors":"Jocelyn Gravel, Chloé Dion, Mandana Fadaei Kermani, Sarah Mousseau, Esli Osmanlliu","doi":"10.1093/pch/pxae062","DOIUrl":null,"url":null,"abstract":"Background: ChatGPT received attention for medical writing. Our objective was to evaluate whether ChatGPT 4.0 could improve the quality of abstracts submitted to a medical conference by clinical researchers.Methods: This was an experimental study involving 24 international researchers (the participants) who provided one original abstract intended for submission at the 2024 Pediatric Academic Society (PAS) conference. We asked ChatGPT-4 to improve the quality of the abstract while adhering to PAS submission guidelines. Participants received the revised version and were tasked with creating a final abstract. The quality of each version (original, ChatGPT and final) was evaluated by the participants themselves using a numeric scale (0-100). Additionally, three co-investigators assessed abstracts blinded to the version. The primary analysis focused on the mean difference in scores between the final and original abstracts.Results: Abstract quality varied between the three versions with mean scores of 82, 65 and 90 for the original, ChatGPT and final versions, respectively. Overall, the final version displayed significantly improved quality compared to the original (mean difference 8.0 points; 95% CI: 5.6-10.3). Independent ratings by the co-investigators confirmed statistically significant improvements (mean difference 1.10 points; 95% CI: 0.54-1.66). Participants identified minor (n = 10) and major (n = 3) factual errors in ChatGPT's abstracts.Conclusion: ChatGPT 4.0 does not produce abstracts of better quality than the one crafted by researchers but it offers suggestions to help them improve their abstracts. It may be more useful for researchers encountering challenges in abstract generation due to limited experience or language barriers.","PeriodicalId":19730,"journal":{"name":"Paediatrics & child health","volume":"30 3","pages":"116-121"},"PeriodicalIF":1.8000,"publicationDate":"2024-09-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12208364/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Paediatrics & child health","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1093/pch/pxae062","RegionNum":4,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/6/1 0:00:00","PubModel":"eCollection","JCR":"Q2","JCRName":"PEDIATRICS","Score":null,"Total":0}

引用次数: 0

Abstract

Background: ChatGPT received attention for medical writing. Our objective was to evaluate whether ChatGPT 4.0 could improve the quality of abstracts submitted to a medical conference by clinical researchers.

Methods: This was an experimental study involving 24 international researchers (the participants) who provided one original abstract intended for submission at the 2024 Pediatric Academic Society (PAS) conference. We asked ChatGPT-4 to improve the quality of the abstract while adhering to PAS submission guidelines. Participants received the revised version and were tasked with creating a final abstract. The quality of each version (original, ChatGPT and final) was evaluated by the participants themselves using a numeric scale (0-100). Additionally, three co-investigators assessed abstracts blinded to the version. The primary analysis focused on the mean difference in scores between the final and original abstracts.

Results: Abstract quality varied between the three versions with mean scores of 82, 65 and 90 for the original, ChatGPT and final versions, respectively. Overall, the final version displayed significantly improved quality compared to the original (mean difference 8.0 points; 95% CI: 5.6-10.3). Independent ratings by the co-investigators confirmed statistically significant improvements (mean difference 1.10 points; 95% CI: 0.54-1.66). Participants identified minor (n = 10) and major (n = 3) factual errors in ChatGPT's abstracts.

Conclusion: ChatGPT 4.0 does not produce abstracts of better quality than the one crafted by researchers but it offers suggestions to help them improve their abstracts. It may be more useful for researchers encountering challenges in abstract generation due to limited experience or language barriers.

查看原文本刊更多论文

ChatGPT-4会提高医学摘要的质量吗？

背景：ChatGPT在医学写作中受到关注。我们的目的是评估ChatGPT 4.0是否可以提高临床研究人员提交给医学会议的摘要的质量。方法：这是一项涉及24名国际研究人员（参与者）的实验性研究，他们提供了一份原始摘要，拟提交给2024年儿科学术学会（PAS）会议。我们要求ChatGPT-4在遵守PAS提交指南的同时提高摘要的质量。参与者收到了修改后的版本，并被要求创建最终摘要。每个版本（原始，ChatGPT和最终版本）的质量由参与者自己使用数字刻度（0-100）进行评估。此外，三位共同研究者评估了对该版本不知情的摘要。初步分析的重点是最终摘要和原始摘要的平均分差异。结果：三个版本的摘要质量存在差异，原始版本、ChatGPT版本和最终版本的平均得分分别为82分、65分和90分。总体而言，最终版本与原始版本相比，质量有了显著提高(平均差8.0分；95% ci: 5.6-10.3)。共同研究者的独立评分证实了统计学上显著的改善(平均差1.10分；95% ci: 0.54-1.66)。参与者确定了ChatGPT摘要中的次要（n = 10）和主要（n = 3）事实错误。结论：ChatGPT 4.0产生的摘要质量并不比研究人员制作的摘要高，但它提供了帮助他们改进摘要的建议。它可能对由于经验有限或语言障碍而遇到抽象生成挑战的研究人员更有用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Paediatrics & child health 医学-小儿科

CiteScore

2.10

自引率

5.30%

发文量

208

审稿时长

>12 weeks

期刊介绍： Paediatrics & Child Health (PCH) is the official journal of the Canadian Paediatric Society, and the only peer-reviewed paediatric journal in Canada. Its mission is to advocate for the health and well-being of all Canadian children and youth and to educate child and youth health professionals across the country. PCH reaches 8,000 paediatricians, family physicians and other child and youth health professionals, as well as ministers and officials in various levels of government who are involved with child and youth health policy in Canada.