Making AI Accessible to Social Work Researchers: An Exploratory Analysis of Using ChatGPT to Screen Articles for Systematic and Scoping Reviews

Purpose: Title and abstract screening is one of the most time- and resource-intensive steps in systematic and scoping reviews (SRs); however, artificial intelligence (AI) can accelerate this step without sacrificing methodological rigor. We test ChatGPT-4o's ability to accurately screen citations and present an accessible approach, tailored to social work scholars with limited AI experience. Method: Through prompt engineering, we tested how two vectorization techniques, three algorithms, and class weighting impact ChatGPT-4o's classification accuracy for article inclusion in two SRs compared to models run in a native Python environment (Jupyter Notebook). Results: ChatGPT-4o-generated models were comparable to Jupyter Notebook, with the bag of words or term frequency-inverse document frequency and logistic regression models performing well. Additionally, adjusting for class imbalance improved performance across samples and models. Discussion: This approach is effective, promoting accessibility for researchers and practitioners. We encourage future researchers to replicate and extend the use of chatbots in the screening process.