Does ChatGPT generate fake results? Challenges in retrieving content through ChatGPT

Abstract

Purpose

ChatGPT is a new development in this technological era. This artificial intelligence-based tool responds to individuals’ queries and produces the requested content within seconds. Therefore, it is becoming popular among academics, the research community and library professionals. This study aims to test (through personal interaction with the tool) the authenticity of the ChatGPT’s produced records. Another objective of the research is to check the relevance between the individuals’ queries and the tool’s results. The research also intends to identify the challenges in retrieving information through ChatGPT.

Design/methodology/approach

The five researchers from different countries and organizations experienced ChatGPT by asking questions on more than 70 subjects. The responses were recorded in Notepad and converted into MS Excel and MS Access to standardize and analyze the data. The investigators consulted 11 reputed databases/sources, including Web of Science and Scopus, to assess the authenticity of the data retrieved through ChatGPT.

Findings

The findings confirmed that over 90% of results produced by ChatGPT were fake (the information did not exist in the literature). Similarly, the study sheds light on the discrepancies, such as irrelevant and incomplete information in the data generated by ChatGPT.

Originality/value

This is a unique study that shares the findings based on the different regions’ researchers’ personal experiences with ChatGPT. The researchers covered different subject areas (above 70) while asking questions to ChatGPT. The paper shares implications for researchers, students, faculty members, academic/research organizations and policymakers.