AI-Generated Versus Human Text: Introducing a New Dataset for Benchmarking and Analysis

Abstract

Artificial intelligence (AI) is increasingly embedded in our everyday lives. With the introduction of ChatGPT in November 2022 by OpenAI, people can now ask a bot to generate comprehensive writeups in seconds. This new transformative technology also introduces ethical, safety, and other general concerns. It is important to harness the power of AI to understand whether a body of text is generated by AI or whether it is organically human. In this article, we create and curate a medium-sized dataset of 10 000 records containing both human and machine-generated text and utilize it to train a reliable model to accurately distinguish between the two. First, we use DistilGPT-2 with various inputs to generate machine text. Then, we acquire an equal sample size of human-generated text. All the text is cleaned, explored, and visualized using the uniform manifold approximation and projection (UMAP) dimensionality reduction technique. Finally, the text is transformed into vectors using several techniques, including bag of words, term frequency-inverse document frequency, bidirectional encoder representations from transformer, and neural network-based embeddings. Machine learning experiments are then performed with traditional models such as logistic regression, random forest, and XGBoost, as well as deep learning models such as long short-term memory (LSTM), convolutional neural network (CNN), and CNN-LSTM. Across all vectorization strategies and machine learning algorithms, we measure accuracy, precision, recall, and F1 scores. We also time each exercise. Each model completes its training within an hour, and we observe scores above 90%. We then use the Shapley additive explanations (SHAP) package on machine learning models to explore if and how we can explain the model to further validate results. Lastly, we deploy our TF-IDF Random Forest model to a user-friendly web application using the Streamlit framework, allowing users without coding expertise to interact with the model.