Human Reviewers' Ability to Differentiate Human-Authored or Artificial Intelligence–Generated Medical Manuscripts

IF 6.9 2区医学 Q1 MEDICINE, GENERAL & INTERNAL

Mayo Clinic proceedings Pub Date : 2025-04-01 DOI:10.1016/j.mayocp.2024.08.029

Scott A. Helgeson MD , Patrick W. Johnson BS , Nilaa Gopikrishnan MD , Tapendra Koirala MD , Pablo Moreno-Franco MD , Rickey E. Carter PhD , Zachary S. Quicksall MS , Charles D. Burger MD

{"title":"Human Reviewers' Ability to Differentiate Human-Authored or Artificial Intelligence–Generated Medical Manuscripts","authors":"Scott A. Helgeson MD , Patrick W. Johnson BS , Nilaa Gopikrishnan MD , Tapendra Koirala MD , Pablo Moreno-Franco MD , Rickey E. Carter PhD , Zachary S. Quicksall MS , Charles D. Burger MD","doi":"10.1016/j.mayocp.2024.08.029","DOIUrl":null,"url":null,"abstract":"<div><h3>Objective</h3><div>To assess the ability of humans to differentiate human-authored vs artificial intelligence (AI)–generated medical manuscripts.</div></div><div><h3>Methods</h3><div>This is a prospective randomized survey study from October 1, 2023, to December 1, 2023, from a single academic center. Artificial intelligence–generated medical manuscripts were created using ChatGPT 3.5 and were evaluated alongside randomly selected human-authored manuscripts. Participants, who were blinded from manuscript selection and creation, were randomized to receive three manuscripts that were either human-authored or AI-generated and had to fill out a survey questionnaire after review regarding who authored the manuscript. The primary outcome was accuracy of human reviewers in differentiating manuscript authors. Secondary outcomes were to identify factors that influenced prediction accuracy.</div></div><div><h3>Results</h3><div>Fifty-one physicians were included in the study, including 12 post-doctorates, 19 assistant professors, and 20 associate or full professors. The overall specificity of 55.6% (95% CI, 30.8% to 78.5%), sensitivity of 31.2% (95% CI,11.0% to 58.7%), positive predictive value of 38.5% (95% CI,13.9% to 68.4%) and negative predictive value of 47.6% (95% CI, 25.7% to 70.2%). A stratified analysis of human-authored manuscripts indicated that high-impact factor manuscripts were identified with higher accuracy than low-impact factor ones (<em>P</em>=.037). For individual-level data, neither academic rank nor prior manuscript review experience significantly predicted the accuracy. The frequency of AI interaction was a significant factor, with occasional (odds ratio [OR], 8.20; <em>P</em>=.016), fairly frequent (OR, 7.13; <em>P</em>=.033), and very frequent (OR, 8.36; <em>P</em>=.030) use associated with correct identification. Further analysis revealed no significant predictors among the papers' qualities.</div></div><div><h3>Conclusion</h3><div>Generative AI such as ChatGPT could create medical manuscripts that could not be differentiated from human-authored manuscripts.</div></div>","PeriodicalId":18334,"journal":{"name":"Mayo Clinic proceedings","volume":"100 4","pages":"Pages 622-633"},"PeriodicalIF":6.9000,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Mayo Clinic proceedings","FirstCategoryId":"3","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S0025619624004890","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"MEDICINE, GENERAL & INTERNAL","Score":null,"Total":0}

引用次数: 0

Abstract

Objective

To assess the ability of humans to differentiate human-authored vs artificial intelligence (AI)–generated medical manuscripts.

Methods

This is a prospective randomized survey study from October 1, 2023, to December 1, 2023, from a single academic center. Artificial intelligence–generated medical manuscripts were created using ChatGPT 3.5 and were evaluated alongside randomly selected human-authored manuscripts. Participants, who were blinded from manuscript selection and creation, were randomized to receive three manuscripts that were either human-authored or AI-generated and had to fill out a survey questionnaire after review regarding who authored the manuscript. The primary outcome was accuracy of human reviewers in differentiating manuscript authors. Secondary outcomes were to identify factors that influenced prediction accuracy.

Results

Fifty-one physicians were included in the study, including 12 post-doctorates, 19 assistant professors, and 20 associate or full professors. The overall specificity of 55.6% (95% CI, 30.8% to 78.5%), sensitivity of 31.2% (95% CI,11.0% to 58.7%), positive predictive value of 38.5% (95% CI,13.9% to 68.4%) and negative predictive value of 47.6% (95% CI, 25.7% to 70.2%). A stratified analysis of human-authored manuscripts indicated that high-impact factor manuscripts were identified with higher accuracy than low-impact factor ones (P=.037). For individual-level data, neither academic rank nor prior manuscript review experience significantly predicted the accuracy. The frequency of AI interaction was a significant factor, with occasional (odds ratio [OR], 8.20; P=.016), fairly frequent (OR, 7.13; P=.033), and very frequent (OR, 8.36; P=.030) use associated with correct identification. Further analysis revealed no significant predictors among the papers' qualities.

Conclusion

Generative AI such as ChatGPT could create medical manuscripts that could not be differentiated from human-authored manuscripts.

查看原文本刊更多论文

人类审稿人区分人类撰写或人工智能生成的医学手稿的能力：一项随机调查研究。

目的：评估人类区分人类撰写与人工智能（AI）生成的医学手稿的能力。方法：这是一项前瞻性随机调查研究，时间为2023年10月1日至2023年12月1日，来自单个学术中心。使用ChatGPT 3.5创建人工智能生成的医学手稿，并与随机选择的人类撰写的手稿一起进行评估。参与者在论文选择和创作过程中被蒙蔽，他们被随机分配到三份由人类撰写或人工智能生成的论文，并在审查后填写一份关于谁是论文作者的调查问卷。主要结果是人类审稿人区分稿件作者的准确性。次要结果是确定影响预测准确性的因素。结果：共纳入51名医师，其中博士后12人，助理教授19人，副教授或正教授20人。总特异性为55.6% (95% CI, 30.8% ~ 78.5%)，敏感性为31.2% (95% CI,11.0% ~ 58.7%)，阳性预测值为38.5% (95% CI,13.9% ~ 68.4%)，阴性预测值为47.6% （95% CI, 25.7% ~ 70.2%）。对人工稿件的分层分析表明，高影响因子稿件的识别准确率高于低影响因子稿件（P= 0.037）。对于个人水平的数据，学术等级和先前的审稿经验都不能显著预测准确性。人工智能互动的频率是一个重要因素，偶尔(优势比[OR]， 8.20；P= 0.016)，相当频繁(OR, 7.13；P= 0.033)，且非常频繁(OR, 8.36；P=.030)使用与正确识别相关。进一步的分析显示，论文质量之间没有显著的预测因子。结论：ChatGPT等生成式人工智能可以创建与人类撰写的手稿无法区分的医学手稿。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Mayo Clinic proceedings 医学-医学：内科

CiteScore

16.80

自引率

1.10%

发文量

383

审稿时长

37 days

期刊介绍： Mayo Clinic Proceedings is a premier peer-reviewed clinical journal in general medicine. Sponsored by Mayo Clinic, it is one of the most widely read and highly cited scientific publications for physicians. Since 1926, Mayo Clinic Proceedings has continuously published articles that focus on clinical medicine and support the professional and educational needs of its readers. The journal welcomes submissions from authors worldwide and includes Nobel-prize-winning research in its content. With an Impact Factor of 8.9, Mayo Clinic Proceedings is ranked #20 out of 167 journals in the Medicine, General and Internal category, placing it in the top 12% of these journals. It invites manuscripts on clinical and laboratory medicine, health care policy and economics, medical education and ethics, and related topics.