{"title":"Can artificial intelligence write science? A comparative analysis of human-written and artificial intelligence-generated scientific writings.","authors":"Karim Rizwan Nathani, Ali-Muhammad Nathani, Maliya Delawan, Aleeza Safdar, Mohamad Bydon","doi":"10.3171/2025.4.SPINE25519","DOIUrl":null,"url":null,"abstract":"<p><strong>Objective: </strong>Artificial intelligence (AI) is increasingly capable of academic writing, with large language models such as ChatGPT showing potential to assist or even generate scientific manuscripts. However, concerns remain regarding the quality, reliability, and interpretive capabilities of AI-generated content. The authors' study aimed to compare the quality of a human-written versus an AI-generated scientific manuscript to evaluate the strengths and limitations of AI in the context of academic publishing.</p><p><strong>Methods: </strong>Two manuscripts were developed using identical titles, abstracts, and tables of a simulated analysis: one authored by a physician with multiple publications, and the other generated by ChatGPT-4o. Three independent and blinded reviewers-two human and one AI-assessed each manuscript across five domains: clarity and readability, coherence and flow, technical accuracy, depth, and conciseness and precision. Each category was scored on a 10-point scale, and qualitative feedback was collected to highlight specific strengths and weaknesses. Additionally, all reviewers were asked to deduce authorship of the manuscripts.</p><p><strong>Results: </strong>The AI-generated manuscript scored higher in clarity and readability (mean 9.0 vs 7.2), but lower in technical accuracy (mean 6.3 vs 9.3) and depth (mean 5.5 vs 7.5). However, reviewers noted that the AI version lacked depth, critical analysis, and contextual interpretation. All reviewers accurately identified the authorship of each manuscript and tended to rate the version more favorably when it aligned with their own origin (human or AI); i.e., human reviewers assigned higher scores to the human-written manuscript, while the AI reviewer scored the AI-generated manuscript higher.</p><p><strong>Conclusions: </strong>Although AI models can improve some aspects of scientific writing, particularly clarity and readability, they fall short in critical reasoning and contextual understanding. This reinforces the importance of human authorship and oversight in maintaining the critical analysis and scientific accuracy essential for academic publishing. AI may be used as a complementary tool to support, rather than replace, human-led scientific writing.</p>","PeriodicalId":16562,"journal":{"name":"Journal of neurosurgery. Spine","volume":" ","pages":"1-6"},"PeriodicalIF":3.1000,"publicationDate":"2025-08-22","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of neurosurgery. Spine","FirstCategoryId":"3","ListUrlMain":"https://doi.org/10.3171/2025.4.SPINE25519","RegionNum":2,"RegionCategory":"医学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"CLINICAL NEUROLOGY","Score":null,"Total":0}
引用次数: 0
Abstract
Objective: Artificial intelligence (AI) is increasingly capable of academic writing, with large language models such as ChatGPT showing potential to assist or even generate scientific manuscripts. However, concerns remain regarding the quality, reliability, and interpretive capabilities of AI-generated content. The authors' study aimed to compare the quality of a human-written versus an AI-generated scientific manuscript to evaluate the strengths and limitations of AI in the context of academic publishing.
Methods: Two manuscripts were developed using identical titles, abstracts, and tables of a simulated analysis: one authored by a physician with multiple publications, and the other generated by ChatGPT-4o. Three independent and blinded reviewers-two human and one AI-assessed each manuscript across five domains: clarity and readability, coherence and flow, technical accuracy, depth, and conciseness and precision. Each category was scored on a 10-point scale, and qualitative feedback was collected to highlight specific strengths and weaknesses. Additionally, all reviewers were asked to deduce authorship of the manuscripts.
Results: The AI-generated manuscript scored higher in clarity and readability (mean 9.0 vs 7.2), but lower in technical accuracy (mean 6.3 vs 9.3) and depth (mean 5.5 vs 7.5). However, reviewers noted that the AI version lacked depth, critical analysis, and contextual interpretation. All reviewers accurately identified the authorship of each manuscript and tended to rate the version more favorably when it aligned with their own origin (human or AI); i.e., human reviewers assigned higher scores to the human-written manuscript, while the AI reviewer scored the AI-generated manuscript higher.
Conclusions: Although AI models can improve some aspects of scientific writing, particularly clarity and readability, they fall short in critical reasoning and contextual understanding. This reinforces the importance of human authorship and oversight in maintaining the critical analysis and scientific accuracy essential for academic publishing. AI may be used as a complementary tool to support, rather than replace, human-led scientific writing.
人工智能(AI)在学术写作方面的能力越来越强,ChatGPT等大型语言模型显示出协助甚至生成科学稿件的潜力。然而,关于人工智能生成内容的质量、可靠性和解释能力的担忧仍然存在。作者的研究旨在比较人类撰写的科学手稿和人工智能生成的科学手稿的质量,以评估人工智能在学术出版领域的优势和局限性。方法:使用相同的标题、摘要和模拟分析的表格开发了两份手稿:一份由有多篇出版物的医生撰写,另一份由chatgpt - 40生成。三名独立的盲法审稿人——两名人类审稿人和一名人工智能审稿人——从五个方面评估了每份手稿:清晰度和可读性、连贯性和流畅性、技术准确性、深度、简洁性和精确性。每个类别都以10分制进行评分,并收集定性反馈以突出特定的优势和劣势。此外,所有审稿人都被要求推断手稿的作者。结果:人工智能生成的稿件在清晰度和可读性方面得分较高(平均9.0 vs 7.2),但在技术准确性(平均6.3 vs 9.3)和深度(平均5.5 vs 7.5)方面得分较低。然而,评论者指出,人工智能版本缺乏深度、批判性分析和上下文解释。所有审稿人都能准确地识别出每一份手稿的作者,并且当它与他们自己的来源(人类或人工智能)一致时,他们倾向于对该版本进行更有利的评价;也就是说,人类审稿人给人类写的手稿更高的分数,而人工智能审稿人给人工智能生成的手稿更高的分数。结论:尽管人工智能模型可以提高科学写作的某些方面,特别是清晰度和可读性,但它们在批判性推理和上下文理解方面存在不足。这加强了人类作者和监督的重要性,以保持学术出版所必需的批判性分析和科学准确性。人工智能可以作为辅助工具来支持,而不是取代人类主导的科学写作。
期刊介绍:
Primarily publish original works in neurosurgery but also include studies in clinical neurophysiology, organic neurology, ophthalmology, radiology, pathology, and molecular biology.