Evaluation of artificial ıntelligence use in ankylosing spondylitis with ChatGPT-4: patient and physician perspectives.

Abstract

Introduction: This study aims to evaluate the accuracy and comprehensiveness of the information provided by ChatGPT-4, an artificial intelligence-based system, regarding ankylosing spondylitis (AS) from the perspectives of patients and physicians.

Method: In this cross-sectional study, 75 questions were asked of ChatGPT-4. These were the most frequently asked questions about AS on Google Trends (group 1), and questions derived from ASAS/EULAR recommendations (group 2 and group 3). Group 2 consisted of open-ended questions, and group 3 consisted of case questions. Two expert rheumatologists scored the responses for accuracy and comprehensiveness. A six-point Likert scale was used to assess accuracy, and a three-point scale for completeness.

Results: The accuracy and completeness scores analyzed in this study were found to be 5.32 ± 1.4 and 2.76 ± 0.5 for group 1, 5.36 ± 1.1 and 2.72 ± 0.45 for group 2, and 4.24 ± 1.96 and 2.36 ± 0.63 for group 3, respectively. There was a significant difference in accuracy and completeness scores between the groups (p = 0.044 and p = 0.019). Cohen's kappa coefficient showed excellent agreement with values of 0.88 for accuracy and 0.90 for completeness.

Conclusion: While the responses to questions in groups 1 and 2 were satisfactory in terms of accuracy and comprehensiveness, the responses to complex case questions in group 3 were not sufficient. ChatGPT-4 appears to be a useful resource for patient education, but response mechanisms for complex clinical scenarios need to be improved. Furthermore, the potential for generating false or fabricated data should be considered; therefore, physicians should evaluate and verify responses. Key Points • ChatGPT-4 demonstrated high accuracy and comprehensiveness in addressing the information needs of patients and physicians regarding axial spondyloarthritis (AS), with satisfactory responses particularly for frequently asked questions and questions derived from ASAS/EULAR recommendations. • The quality of responses significantly decreased in the group of' case questions' involving more complex scenarios, indicating that the AI may be inadequate in clinical decision-making processes. • ChatGPT-4 can facilitate access to health information for patients, potentially reducing unnecessary clinic visits and serving as a primary source of information for those with limited access to healthcare professionals, thereby enhancing the overall effectiveness of patient care. • While ChatGPT-4 shows promise as a valuable tool for patient education, there is a clear need for the development of its response mechanisms, particularly for complex clinical scenarios.