Artificial intelligence in asthma health literacy: a comparative analysis of ChatGPT versus Gemini.

Abstract

Background: Asthma is a complex and heterogeneous chronic disease affecting over 300 million individuals worldwide. Despite advances in pharmacotherapy, poor disease control remains a major challenge, necessitating innovative approaches to patient education and self-management. Artificial intelligence driven chatbots, such as ChatGPT and Gemini, have the potential to enhance asthma care by providing real-time, evidence-based information. As asthma management moves toward personalized medicine, AI could support individualized education and treatment guidance. However, concerns remain regarding the accuracy and reliability of AI-generated medical content.

Objective: This study evaluated the accuracy of ChatGPT (version 4.0) and Gemini (version 1.2) in providing asthma-related health information using the Patient-completed Asthma Knowledge Questionnaire, a validated asthma literacy tool.

Methods: A cross-sectional study was conducted in which both AI models answered 54 standardized asthma-related items. Responses were classified as correct or incorrect based on alignment with validated clinical knowledge. Accuracy was assessed using descriptive statistics, Cohen's kappa for inter-model agreement, and chi-square tests for comparative performance.

Results: ChatGPT achieved an accuracy of 96.3% (52/54 correct; 95% CI: 87.5%-99.0%), while Gemini scored 92.6% (50/54 correct; 95% CI: 82.5%-97.1%), with no statistically significant difference (p = 0.67). Cohen's kappa demonstrated near-perfect agreement for ChatGPT (κ = 0.91) and strong agreement for Gemini (κ = 0.82).

Conclusion: ChatGPT and Gemini demonstrated high accuracy in delivering asthma-related health information, supporting their potential as adjunct tools for patient education. AI models could potentially play a role in personalized asthma management by providing tailored treatment guidance and improving patient engagement.