Global variations in artificial intelligence-generated information on juvenile idiopathic arthritis.

Abstract

Objectives: We aimed to evaluate similarities and variations of information provided by Large Language Models (LLMs) across diverse world regions by analyzing responses to validated questions on oligoarticular juvenile idiopathic arthritis (oJIA).

Methods: The ten PICOs related to the oJIA treatment on the 2021 American College of Rheumatology recommendations were simultaneously prompted in English to ChatGPT 4o from five different countries (Canada, India, Italy, Kenya and Türkiye). Readability was assessed through the Flesch Reading Ease Score (FRES), distinctiveness of terms through the Term Frequency-Inverse Document Frequency (TF-IDF) analysis. Co-occurrence networks (CONs) detailed the relationships between terms. Three experts rated the adherence of responses to recommendations using a Likert-like scale.

Results: All the responses were difficult or very difficult to read, with a median FRES of 30 [24-34]. Depending on the expert, 52% to 84% of responses were mostly or fully adherent to the recommendations, with similar adherence rates across countries. No response was not adherent at all. Inter-rater agreement on the adherence of LLM-generated responses was generally weak (Kappa values mostly below 0.40), highlighting the challenges of consistently evaluating AI-generated medical information. The TF-IDF analysis showed that the distinctiveness of terminology in LLM-generated responses varied across countries, with scores ranging from 0.60-0.85. CONs detailed a strong focus on intra-articular corticosteroid treatments in Italy and emphasis on short- and long-term outcomes in Kenya.

Conclusion: LLM-generated content should be critically evaluated in clinical practice, especially in the context of regional differences.