QUALITY AND READABILITY OF CHATGPT IN ANSWERING PATIENTS’ PREVENTIVE CARDIOLOGY QUESTIONS: A PILOT STUDY

Therapeutic Area

Other

Background

As artificial intelligence (AI) becomes increasingly integrated into healthcare, ChatGPT has emerged as a promising tool for patient education. However, research on its suitability for preventive cardiology remains limited. With patients increasingly relying on online health information, it is essential that content is both scientifically accurate and accessible to individuals of all health literacy levels. This study evaluates the quality and readability of ChatGPT’s responses to common questions on lifestyle modification, women’s cardiovascular health, and cholesterol management.

Methods

Twenty-six questions (8 on lifestyle modifications, 8 on women’s cardiovascular health, and 10 on cholesterol management) were queried using the GPT-4 model. Responses were independently evaluated by three board-certified preventive cardiologists, referencing the latest national cardiovascular guidelines. Quality was assessed using a 5-point Likert scale for correctness, comprehensiveness, conciseness, and comprehensibility, previously employed in medical AI research. Readability was analyzed using the Flesch-Kincaid Grade Level and other standardized readability metrics.

Results

ChatGPT provided adequate responses to 88.4% (23/26) of questions, with mean (SE) scores of 3.71 ± 0.20 for correctness, 4.06 ± 0.14 for conciseness, 4.06 ± 0.13 for comprehensiveness, and 4.40 ± 0.10 for comprehensibility. The highest-scoring topic was lifestyle modification (84.4%), followed by cholesterol management (81.2%) and women’s cardiovascular health (77.8%). Among inadequate responses, key limitations included overstating the risks of low LDL cholesterol and exaggerating the benefits of estrogen replacement therapy (ERT) for postmenopausal CVD risk reduction. ChatGPT also provided unsupported recommendations regarding dietary supplements for CVD prevention. Readability analysis revealed responses at a 13th-grade level, exceeding the recommended 6th-grade level for patient education.

Conclusions

ChatGPT’s responses were generally suitable for topics such as heart-healthy diets, exercise, weight management, epidemiology and clinical presentation of CVD in women, postmenopausal CVD risk, cholesterol-lowering therapy, statin-associated side effects, and Lp(a) risk stratification. However, inaccuracies persisted in dietary supplementation, ERT, and very low LDL levels. Enhancements in AI training are needed to improve accuracy in these areas. Additionally, the high readability level limits accessibility for the general public, underscoring the need for optimization to ensure clear and reliable patient education.