Daphne E DeTemple, Simon Störzer, Sahar Arbabzadah, Anna Riddermann, Felix Gronau, Kai Timrott, Hüseyin Bektas, Moritz Kleine
{"title":"The role of large language models in the writing of surgical reviews: fact or fantasy?","authors":"Daphne E DeTemple, Simon Störzer, Sahar Arbabzadah, Anna Riddermann, Felix Gronau, Kai Timrott, Hüseyin Bektas, Moritz Kleine","doi":"10.1007/s00423-026-04095-2","DOIUrl":null,"url":null,"abstract":"<p><strong>Background: </strong>Amidst the current enthusiasm concerning artificial intelligence and its possible application in the composition of different kinds of scientific and non-scientific written documents, we evaluated the usage of artificial intelligence for writing surgical short reviews.</p><p><strong>Methods: </strong>In order to assess the formal and content quality of AI-generated texts compared to human written texts, ten AI-based text generators (five chatbots and five content creators) and four surgeons in training received the same prompt for a short scientific article on a liver surgery theme. All texts were anonymized and subsequently evaluated by three experienced liver surgeons based on a pre-defined scoring scheme, as well as for quality of references and readability according to readability indices. Furthermore, all texts were tested for plagiarism using PlagScan.</p><p><strong>Results: </strong>Overall percentage of correct assessment for AI/non-AI generation by experienced surgeons lay at 78.57%. Human written text had a mean word count of 1054 versus 874 in AI-generated text, with a higher mean Flesh Reading Ease Score (FRE, 26.2 ± 5.1 versus 17.7 ± 6.1). References were PubMed-listed in 100% for human versus 46% for AI-generated text, with only one non-human text reaching 100% formally correct citation of references. PlagScan found 6.4%±1.3 mean resemblance to existing texts for human versus 7.6%±4.5 for AI-generated text.</p><p><strong>Discussion: </strong>Overall, AI could already mislead experienced scientific surgeons in 26.7% of cases into believing it to be human. However, formal requirements, especially considering referencing, are still in great need of improvement with only one of AI-generated articles fulfilling our quality requirements.</p>","PeriodicalId":17983,"journal":{"name":"Langenbeck's Archives of Surgery","volume":"411 1","pages":""},"PeriodicalIF":2.2000,"publicationDate":"2026-07-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC13331823/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Langenbeck's Archives of Surgery","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1007/s00423-026-04095-2","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"SURGERY","Score":null,"Total":0}
引用次数: 0
Abstract
Background: Amidst the current enthusiasm concerning artificial intelligence and its possible application in the composition of different kinds of scientific and non-scientific written documents, we evaluated the usage of artificial intelligence for writing surgical short reviews.
Methods: In order to assess the formal and content quality of AI-generated texts compared to human written texts, ten AI-based text generators (five chatbots and five content creators) and four surgeons in training received the same prompt for a short scientific article on a liver surgery theme. All texts were anonymized and subsequently evaluated by three experienced liver surgeons based on a pre-defined scoring scheme, as well as for quality of references and readability according to readability indices. Furthermore, all texts were tested for plagiarism using PlagScan.
Results: Overall percentage of correct assessment for AI/non-AI generation by experienced surgeons lay at 78.57%. Human written text had a mean word count of 1054 versus 874 in AI-generated text, with a higher mean Flesh Reading Ease Score (FRE, 26.2 ± 5.1 versus 17.7 ± 6.1). References were PubMed-listed in 100% for human versus 46% for AI-generated text, with only one non-human text reaching 100% formally correct citation of references. PlagScan found 6.4%±1.3 mean resemblance to existing texts for human versus 7.6%±4.5 for AI-generated text.
Discussion: Overall, AI could already mislead experienced scientific surgeons in 26.7% of cases into believing it to be human. However, formal requirements, especially considering referencing, are still in great need of improvement with only one of AI-generated articles fulfilling our quality requirements.
期刊介绍：
Langenbeck''s Archives of Surgery aims to publish the best results in the field of clinical surgery and basic surgical research. The main focus is on providing the highest level of clinical research and clinically relevant basic research. The journal, published exclusively in English, will provide an international discussion forum for the controlled results of clinical surgery. The majority of published contributions will be original articles reporting on clinical data from general and visceral surgery, while endocrine surgery will also be covered. Papers on basic surgical principles from the fields of traumatology, vascular and thoracic surgery are also welcome. Evidence-based medicine is an important criterion for the acceptance of papers.
求助内容：
应助结果提醒方式：
京公网安备 11010802042870号
