Jonathan M Carnino, Nicholas Y K Chong, Henry Bayly, Lindsay R Salvati, Hardeep S Tiwana, Jessica R Levi
{"title":"AI-generated text in otolaryngology publications: a comparative analysis before and after the release of ChatGPT.","authors":"Jonathan M Carnino, Nicholas Y K Chong, Henry Bayly, Lindsay R Salvati, Hardeep S Tiwana, Jessica R Levi","doi":"10.1007/s00405-024-08834-3","DOIUrl":null,"url":null,"abstract":"<p><strong>Purpose: </strong>This study delves into the broader implications of artificial intelligence (AI) text generation technologies, including large language models (LLMs) and chatbots, on the scientific literature of otolaryngology. By observing trends in AI-generated text within published otolaryngology studies, this investigation aims to contextualize the impact of AI-driven tools that are reshaping scientific writing and communication.</p><p><strong>Methods: </strong>Text from 143 original articles published in JAMA Otolaryngology - Head and Neck Surgery was collected, representing periods before and after ChatGPT's release in November 2022. The text from each article's abstract, introduction, methods, results, and discussion were entered into ZeroGPT.com to estimate the percentage of AI-generated content. Statistical analyses, including T-Tests and Fligner-Killeen's tests, were conducted using R.</p><p><strong>Results: </strong>A significant increase was observed in the mean percentage of AI-generated text post-ChatGPT release, especially in the abstract (from 34.36 to 46.53%, p = 0.004), introduction (from 32.43 to 45.08%, p = 0.010), and discussion sections (from 15.73 to 25.03%, p = 0.015). Publications of authors from non-English speaking countries demonstrated a higher percentage of AI-generated text.</p><p><strong>Conclusion: </strong>This study found that the advent of ChatGPT has significantly impacted writing practices among researchers publishing in JAMA Otolaryngology - Head and Neck Surgery, raising concerns over the accuracy of AI-created content and potential misinformation risks. This manuscript highlights the evolving dynamics between AI technologies, scientific communication, and publication integrity, emphasizing the urgent need for continued research in this dynamic field. The findings also suggest an increasing reliance on AI tools like ChatGPT, raising questions about their broader implications for scientific publishing.</p>","PeriodicalId":11952,"journal":{"name":"European Archives of Oto-Rhino-Laryngology","volume":" ","pages":"6141-6146"},"PeriodicalIF":1.9000,"publicationDate":"2024-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC11513233/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"European Archives of Oto-Rhino-Laryngology","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.1007/s00405-024-08834-3","RegionNum":3,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2024/7/17 0:00:00","PubModel":"Epub","JCR":"Q2","JCRName":"OTORHINOLARYNGOLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Purpose: This study delves into the broader implications of artificial intelligence (AI) text generation technologies, including large language models (LLMs) and chatbots, on the scientific literature of otolaryngology. By observing trends in AI-generated text within published otolaryngology studies, this investigation aims to contextualize the impact of AI-driven tools that are reshaping scientific writing and communication.
Methods: Text from 143 original articles published in JAMA Otolaryngology - Head and Neck Surgery was collected, representing periods before and after ChatGPT's release in November 2022. The text from each article's abstract, introduction, methods, results, and discussion were entered into ZeroGPT.com to estimate the percentage of AI-generated content. Statistical analyses, including T-Tests and Fligner-Killeen's tests, were conducted using R.
Results: A significant increase was observed in the mean percentage of AI-generated text post-ChatGPT release, especially in the abstract (from 34.36 to 46.53%, p = 0.004), introduction (from 32.43 to 45.08%, p = 0.010), and discussion sections (from 15.73 to 25.03%, p = 0.015). Publications of authors from non-English speaking countries demonstrated a higher percentage of AI-generated text.
Conclusion: This study found that the advent of ChatGPT has significantly impacted writing practices among researchers publishing in JAMA Otolaryngology - Head and Neck Surgery, raising concerns over the accuracy of AI-created content and potential misinformation risks. This manuscript highlights the evolving dynamics between AI technologies, scientific communication, and publication integrity, emphasizing the urgent need for continued research in this dynamic field. The findings also suggest an increasing reliance on AI tools like ChatGPT, raising questions about their broader implications for scientific publishing.
期刊介绍:
Official Journal of
European Union of Medical Specialists – ORL Section and Board
Official Journal of Confederation of European Oto-Rhino-Laryngology Head and Neck Surgery
"European Archives of Oto-Rhino-Laryngology" publishes original clinical reports and clinically relevant experimental studies, as well as short communications presenting new results of special interest. With peer review by a respected international editorial board and prompt English-language publication, the journal provides rapid dissemination of information by authors from around the world. This particular feature makes it the journal of choice for readers who want to be informed about the continuing state of the art concerning basic sciences and the diagnosis and management of diseases of the head and neck on an international level.
European Archives of Oto-Rhino-Laryngology was founded in 1864 as "Archiv für Ohrenheilkunde" by A. von Tröltsch, A. Politzer and H. Schwartze.