Human or Machine? A Comparative Analysis of Artificial Intelligence-Generated Writing Detection in Personal Statements.

Abstract

Introduction: This study examines the ability of human readers, recurrence quantification analysis (RQA), and an online artificial intelligence (AI) detection tool (GPTZero) to distinguish between AI-generated and human-written personal statements in physical therapist education program applications.

Review of literature: The emergence of large language models such as ChatGPT and Google Gemini has raised concerns about the authenticity of personal statements. Previous studies have reported varying degrees of success in detecting AI-generated text.

Subjects: Data were collected from 50 randomly selected nonmatriculated individuals who applied to the Mayo Clinic School of Health Sciences Doctor of Physical Therapy Program during the 2021-2022 application cycle.

Methods: Fifty personal statements from applicants were pooled with 50 Google Gemini-generated statements, then analyzed by 2 individuals, RQA, and GPTZero. RQA provided quantitative measures of lexical sophistication, whereas GPTZero used advanced machine learning algorithms to quantify AI-specific text characteristics.

Results: Human raters demonstrated high agreement (κ = 0.92) and accuracy (97% and 99%). RQA parameters, particularly recurrence and max line, differentiated human- from AI-generated statements (areas under receiver operating characteristic [ROC] curve = 0.768 and 0.859, respectively). GPTZero parameters including simplicity, perplexity, and readability also differentiated human- from AI-generated statements (areas under ROC curve > 0.875).

Discussion and conclusion: The study reveals that human raters, RQA, and GPTZero offer varying levels of accuracy in differentiating human-written from AI-generated personal statements. The findings could have important implications in academic admissions processes, where distinguishing between human- and AI-generated submissions is becoming increasingly important. Future research should explore integrating these methods to enhance the robustness and reliability of personal statement content evaluation across various domains. Three strategies for managing AI's role in applications-for applicants, governing organizations, and academic institutions-are provided to promote integrity and accountability in admission processes.