Artificial intelligence in cardiology education: A performance comparison analysis of large language models

Abstract

Introduction and objectives

Artificial intelligence (AI) and large language models (LLMs) are being progressively integrated into medical practice. This study aimed to evaluate 7 different LLMs by testing their responses to cardiology-related questions based on European Society of Cardiology (ESC) guidelines.

Methods

Cardiology-related questions were formulated using 6 ESC guidelines documents. Multiple-choice questions (MCQs) were created to assess the model's accuracy. The questions were repeated across 3 separate rounds. We also compared general LLM scores with 3 cardiology residents preparing for the ESC European Exam in Core Cardiology.

Results

A total of 1096 MCQs were generated. Grok emerged as the top-performing model with a mean accuracy of 85.27%, significantly outperforming Gemini (81.77%; P < .001), Le Chat (81.60%; P = .016), Perplexity (82.17%; P = .034), ChatGPT (82.86%; P = .018) and DeepSeek (84.13%; P = .032). All models showed consistent improvement over time, with Grok and Le Chat showing faster performance enhancement (Friedman test, P < .001). LLM accuracy was comparable to that of the 3 cardiology residents across all topic groups, with no statistically significant differences (P = .853, P = .691, P = .061).

Conclusions

All LLMs exhibit promising accuracy and continuous improvement when answering cardiology-related questions with performances like senior cardiology residents, highlighting their potential for real-world integration in medical practice although. Overall, Grok performed better than the other LLMs. As physicians increasingly incorporate LLMs into clinical decision-making, attention must be paid to the models’ limitations, particularly their lack of self-awareness.