The Use of Generative AI for Scientific Literature Searches for Systematic Reviews: ChatGPT and Microsoft Bing AI Performance Evaluation

Abstract

Background: A large language model (LLM) is a type of artificial intelligence (AI) model that opens up great possibilities for healthcare practice, research, and education, although scholars have highlighted that there is a need to proactively address current issues regarding its use. One of the best-known LLMs is ChatGPT. Objective: This study aims to explore the potential of ChatGPT as a real-time literature search tool for systematic reviews and clinical decision support system (CDSS). Methods: The search results of a systematic review study on the treatment of Peyronie's Disease published by human experts were selected as a benchmark, and the literature search formula of the study was applied to ChatGPT and Microsoft Bing to compare with human researchers. To determine the accuracy of the retrieved literature, we graded it as A, B, C, and F for only those cases where actual literature exists. Results: The benchmark human researcher's randomized controlled trial search results were 24. ChatGPT collected 1287 literature search results through 639 questions, and 7 of them were exactly matched, and Microsoft Bing collected 48 literature search results through 223 questions, and 19 of them were exactly matched with human search results. Conclusions: This is the first study to compare artificial intelligence (AI) and conventional human systematic review methods as a real-time literature collection tool for evidence-based medicine. The results suggest that the use of ChatGPT as a tool for real-time evidence generation is not yet accurate and feasible. Therefore, researchers should be cautious about using such AI.