Large language model content detectors: Effects of human editing and ChatGPT mimicking individual style-A preliminary study.

The use of large language model (LLM), such as ChatGPT, in academic writing is increasing, but the extent to which LLM-generated content can evade detection remains unclear. This descriptive pilot study investigates whether LLM-generated abstracts, edited by humans or LLM trained to mimic a specific writing style, can escape LLM detectors. Using a previously published original article, ChatGPT-4 generated an abstract (Abstract 1). This abstract underwent three modifications: context-based human editing (Abstract 2), stylistic human editing (Abstract 3), and ChatGPT editing incorporating the author's writing style (Abstract 4). The genuine human-written abstract from the original article served as Abstract 5. Five freely available LLM detectors analyzed these abstracts, providing LLM-generated probability scores. The genuinely LLM-generated manuscript (Abstract 1) was judged as LLM-generated with 82%-100% (median: 100%) probability. The genuinely human-written manuscript (Abstract 5) was judged as human-written with the LLM-generated probability of 0%-13% (median: 0%). Human-edited abstracts (Abstracts 2 and 3) exhibited a decreasing LLM-generated probability 4%-71% (median: 64%) and 2%-65% (median: 61%), respectively, but varied widely among detectors. The LLM-mimicked abstract (Abstract 4) was classified as LLM-generated, with LLM-generated probability ranging 82%-100% (median: 100%). The results showed variations across different LLM detectors. Supplementary experiments demonstrated a similar trend. Human editing reduces LLM-detection probabilities but does not guarantee evasion. LLM-generated content mimicking a specific writing style remains largely detectable. This preliminary experiment provided a novel study concept. Further studies on various manuscripts and different LLM detection methods will enhance understanding of the relationship between LLM-aided paper writing and LLM detectors.