Evaluating the Detection Accuracy of AI-Generated Content in Plastic Surgery: A Comparative Study of Medical Professionals and AI Tools.

IF 3.2 2区医学 Q1 SURGERY

Plastic and reconstructive surgery Pub Date : 2025-06-10 DOI:10.1097/PRS.0000000000012242

Keenan S Fine, Emily E Zona, Aidan W O'Shea, Ellen C Shaffrey, Pradeep K Attaluri, Peter J Wirth, Aaron M Dingle, Samuel Poore

{"title":"Evaluating the Detection Accuracy of AI-Generated Content in Plastic Surgery: A Comparative Study of Medical Professionals and AI Tools.","authors":"Keenan S Fine, Emily E Zona, Aidan W O'Shea, Ellen C Shaffrey, Pradeep K Attaluri, Peter J Wirth, Aaron M Dingle, Samuel Poore","doi":"10.1097/PRS.0000000000012242","DOIUrl":null,"url":null,"abstract":"Background: The growing use of artificial intelligence (AI) in academic writing raises concerns about the integrity of scientific manuscripts and the ability to accurately distinguish human-written from AI-generated content. This study evaluates the ability of medical professionals and AI-detection tools to identify AI involvement in plastic surgery manuscripts.Methods: Eight manuscript passages across four topics were assessed, with four on plastic surgery. Passages were human-written, human-written with AI edits, or fully AI-generated. Twenty-four raters, including medical students, residents, and attendings, classified the passages by origin. Interrater reliability was measured using Fleiss' kappa. Human-written and AI-generated manuscripts were analyzed using three different online AI detection tools. A receiver operator curve (ROC) analysis was conducted to assess their accuracy in detecting AI-generated content. Intraclass correlation coefficients (ICC) were calculated to assess the agreement among the detection tools; these tools identify AI-generated content within the passages in terms of percentage generated by AI.Results: Raters correctly identified the origin of passages 26.5% of the time. For AI-generated passages, accuracy was 34.4%, and for human-written passages, 14.5% (p=0.012). Interrater reliability was poor (kappa=0.078). AI detection tools showed strong discriminatory power (AUC=0.962), but false-positives were frequent at optimal cutoffs (25%-50%). The intraclass correlation coefficient (ICC) between tools was low (-0.118).Conclusions: Medical professionals and AI detection tools struggle to reliably identify AI-generated content. While AI tools demonstrated high discriminatory power, they often misclassified human-written passages. These findings highlight the need for improved methods to protect the integrity of scientific writing and prevent false plagiarism claims.","PeriodicalId":20128,"journal":{"name":"Plastic and reconstructive surgery","volume":" ","pages":""},"PeriodicalIF":3.2000,"publicationDate":"2025-06-10","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Plastic and reconstructive surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1097/PRS.0000000000012242","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q1","JCRName":"SURGERY","Score":null,"Total":0}

引用次数: 0

Abstract

Background: The growing use of artificial intelligence (AI) in academic writing raises concerns about the integrity of scientific manuscripts and the ability to accurately distinguish human-written from AI-generated content. This study evaluates the ability of medical professionals and AI-detection tools to identify AI involvement in plastic surgery manuscripts.

Methods: Eight manuscript passages across four topics were assessed, with four on plastic surgery. Passages were human-written, human-written with AI edits, or fully AI-generated. Twenty-four raters, including medical students, residents, and attendings, classified the passages by origin. Interrater reliability was measured using Fleiss' kappa. Human-written and AI-generated manuscripts were analyzed using three different online AI detection tools. A receiver operator curve (ROC) analysis was conducted to assess their accuracy in detecting AI-generated content. Intraclass correlation coefficients (ICC) were calculated to assess the agreement among the detection tools; these tools identify AI-generated content within the passages in terms of percentage generated by AI.

Results: Raters correctly identified the origin of passages 26.5% of the time. For AI-generated passages, accuracy was 34.4%, and for human-written passages, 14.5% (p=0.012). Interrater reliability was poor (kappa=0.078). AI detection tools showed strong discriminatory power (AUC=0.962), but false-positives were frequent at optimal cutoffs (25%-50%). The intraclass correlation coefficient (ICC) between tools was low (-0.118).

Conclusions: Medical professionals and AI detection tools struggle to reliably identify AI-generated content. While AI tools demonstrated high discriminatory power, they often misclassified human-written passages. These findings highlight the need for improved methods to protect the integrity of scientific writing and prevent false plagiarism claims.

查看原文本刊更多论文

评估整形外科中人工智能生成内容的检测准确性：医疗专业人员和人工智能工具的比较研究。

背景：人工智能（AI）在学术写作中的使用越来越多，引发了人们对科学手稿完整性的担忧，以及准确区分人类写作和人工智能生成内容的能力。本研究评估了医疗专业人员和人工智能检测工具识别人工智能参与整形手术手稿的能力。方法：对四个主题的八篇稿件进行评估，其中四篇是关于整形外科的。段落是人工写的，人工写的，人工编辑的，或者完全由人工智能生成的。包括医学生、住院医生和主治医生在内的24名评判员按来源对这些段落进行了分类。信度采用Fleiss kappa法测量。使用三种不同的在线人工智能检测工具分析了人类撰写和人工智能生成的手稿。进行了受试者操作曲线（ROC）分析，以评估它们在检测人工智能生成内容方面的准确性。计算类内相关系数（ICC）来评估检测工具之间的一致性；这些工具根据AI生成的百分比来识别段落中AI生成的内容。结果：评分者正确识别传代来源的准确率为26.5%。对于人工智能生成的文章，准确率为34.4%，对于人类编写的文章，准确率为14.5% （p=0.012）。量表间信度差（kappa=0.078）。人工智能检测工具表现出较强的判别能力（AUC=0.962），但在最佳截止点（25% ~ 50%）出现假阳性的频率较高。工具间的类内相关系数（ICC）较低（-0.118）。结论：医疗专业人员和人工智能检测工具难以可靠地识别人工智能生成的内容。虽然人工智能工具显示出很强的歧视性，但它们经常对人类写的文章进行错误分类。这些发现强调需要改进方法来保护科学写作的完整性，防止虚假的剽窃声明。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Plastic and reconstructive surgery 医学-外科

CiteScore

5.00

自引率

13.90%

发文量

1436

审稿时长

1.5 months

期刊介绍： For more than 70 years Plastic and Reconstructive Surgery® has been the one consistently excellent reference for every specialist who uses plastic surgery techniques or works in conjunction with a plastic surgeon. Plastic and Reconstructive Surgery® , the official journal of the American Society of Plastic Surgeons, is a benefit of Society membership, and is also available on a subscription basis. Plastic and Reconstructive Surgery® brings subscribers up-to-the-minute reports on the latest techniques and follow-up for all areas of plastic and reconstructive surgery, including breast reconstruction, experimental studies, maxillofacial reconstruction, hand and microsurgery, burn repair, cosmetic surgery, as well as news on medicolegal issues. The cosmetic section provides expanded coverage on new procedures and techniques and offers more cosmetic-specific content than any other journal. All subscribers enjoy full access to the Journal''s website, which features broadcast quality videos of reconstructive and cosmetic procedures, podcasts, comprehensive article archives dating to 1946, and additional benefits offered by the newly-redesigned website.