The ChatGPT conundrum: Human-generated scientific manuscripts misidentified as AI creations by AI text detection tool

Q2 Medicine

Journal of Pathology Informatics Pub Date : 2023-01-01 DOI:10.1016/j.jpi.2023.100342

Hooman H. Rashidi , Brandon D. Fennell , Samer Albahra , Bo Hu , Tom Gorbett

{"title":"The ChatGPT conundrum: Human-generated scientific manuscripts misidentified as AI creations by AI text detection tool","authors":"Hooman H. Rashidi , Brandon D. Fennell , Samer Albahra , Bo Hu , Tom Gorbett","doi":"10.1016/j.jpi.2023.100342","DOIUrl":null,"url":null,"abstract":"<div><p>AI Chat Bots such as ChatGPT are revolutionizing our AI capabilities, especially in text generation, to help expedite many tasks, but they introduce new dilemmas. The detection of AI-generated text has become a subject of great debate considering the AI text detector’s known and unexpected limitations. Thus far, much research in this area has focused on the detection of AI-generated text; however, the goal of this study was to evaluate the opposite scenario, an AI-text detection tool's ability to discriminate human-generated text. Thousands of abstracts from several of the most well-known scientific journals were used to test the predictive capabilities of these detection tools, assessing abstracts from 1980 to 2023. We found that the AI text detector erroneously identified up to 8% of the known real abstracts as AI-generated text. This further highlights the current limitations of such detection tools and argues for novel detectors or combined approaches that can address this shortcoming and minimize its unanticipated consequences as we navigate this new AI landscape.</p></div>","PeriodicalId":37769,"journal":{"name":"Journal of Pathology Informatics","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.sciencedirect.com/science/article/pii/S2153353923001566/pdfft?md5=41b1d6a95706d03ee5115451f14d73c3&pid=1-s2.0-S2153353923001566-main.pdf","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Pathology Informatics","FirstCategoryId":"1085","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2153353923001566","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"Medicine","Score":null,"Total":0}

引用次数: 0

Abstract

AI Chat Bots such as ChatGPT are revolutionizing our AI capabilities, especially in text generation, to help expedite many tasks, but they introduce new dilemmas. The detection of AI-generated text has become a subject of great debate considering the AI text detector’s known and unexpected limitations. Thus far, much research in this area has focused on the detection of AI-generated text; however, the goal of this study was to evaluate the opposite scenario, an AI-text detection tool's ability to discriminate human-generated text. Thousands of abstracts from several of the most well-known scientific journals were used to test the predictive capabilities of these detection tools, assessing abstracts from 1980 to 2023. We found that the AI text detector erroneously identified up to 8% of the known real abstracts as AI-generated text. This further highlights the current limitations of such detection tools and argues for novel detectors or combined approaches that can address this shortcoming and minimize its unanticipated consequences as we navigate this new AI landscape.

查看原文本刊更多论文

ChatGPT难题:人工智能文本检测工具错误地将人类生成的科学手稿识别为人工智能创作

像ChatGPT这样的人工智能聊天机器人正在彻底改变我们的人工智能能力，特别是在文本生成方面，以帮助加快许多任务，但它们引入了新的困境。考虑到人工智能文本检测器已知的和意想不到的局限性，人工智能生成文本的检测已经成为一个非常有争议的话题。到目前为止，这一领域的许多研究都集中在人工智能生成文本的检测上;然而，本研究的目的是评估相反的情况，即人工智能文本检测工具区分人类生成文本的能力。研究人员使用了来自几家最知名科学期刊的数千篇摘要来测试这些检测工具的预测能力，评估了1980年至2023年的摘要。我们发现，人工智能文本检测器错误地将多达8%的已知真实摘要识别为人工智能生成的文本。这进一步强调了这些检测工具目前的局限性，并提出了新的检测器或组合方法，可以解决这一缺点，并在我们导航新的人工智能领域时最大限度地减少其意想不到的后果。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Journal of Pathology Informatics Medicine-Pathology and Forensic Medicine

CiteScore

3.70

自引率

0.00%

发文量

审稿时长

18 weeks

期刊介绍： The Journal of Pathology Informatics (JPI) is an open access peer-reviewed journal dedicated to the advancement of pathology informatics. This is the official journal of the Association for Pathology Informatics (API). The journal aims to publish broadly about pathology informatics and freely disseminate all articles worldwide. This journal is of interest to pathologists, informaticians, academics, researchers, health IT specialists, information officers, IT staff, vendors, and anyone with an interest in informatics. We encourage submissions from anyone with an interest in the field of pathology informatics. We publish all types of papers related to pathology informatics including original research articles, technical notes, reviews, viewpoints, commentaries, editorials, symposia, meeting abstracts, book reviews, and correspondence to the editors. All submissions are subject to rigorous peer review by the well-regarded editorial board and by expert referees in appropriate specialties.