A Comparative Study of the Accuracy of Turn-It-In’s Artificial Intelligence Detector in CTU Doctoral Assignments

Abstract

As access to artificial intelligence becomes mainstream in college paper writing, universities must find concrete methods of verifying work submitted as a student’s original content was not significantly developed using AI. In response to these potential academic integrity issues, plagiarism detectors, like TurnItIn, have developed artificial intelligence detectors that provide the percentage of a student submission identified as written by a computer. TurnItIn (TII) claims a 98% confidence level concerning its accuracy. In this study, 48 papers were randomly selected from the Colorado Technical University’s Doctoral Studies courses. These papers were written before the public release of ChatGPT 3.5. Papers were scored for similarity and AI-generated content through TII. These same papers were then edited for grammar and clarity using ChatGPT, and the similarity and AI-generated content scores were recalculated through TII. Wilcoxon Ranked-Sum tests were conducted to determine if a statistical difference occurred between the original and updated similarity and AI-generated content scores. The results demonstrate that TII’s confidence level may not be as accurate as claimed and that the detection of AI-generated content requires further testing before being used to determine if an act of academic dishonesty took place.