Evaluation of ChatGPT’s responses to frequently asked questions about shoulder arthroplasty

Background

Patients now have access to numerous resources about common orthopedic procedures before clinical consultations. Recently, artificial intelligence (AI)-powered chatbots have gained popularity, offering human-like, convincing responses. A prominent example is ChatGPT, an AI-based chat technology, which has seen rapid growth in acceptance. Given the increasing likelihood that patients will turn to this technology for educational information prior to undergoing surgical intervention, we aimed to determine ChatGPT’s ability to provide adequate answers to frequently asked questions about shoulder arthroplasty.

Methods

After a literature review, we formulated the 10 most frequently asked questions about shoulder arthroplasty and presented them to the chatbot without follow-up questions or repetitions. The responses were analyzed by 15 independent shoulder surgeons, who evaluated each response for accuracy using an evidence-based approach. The responses were classified as “excellent response without need for clarification,” “satisfactory requiring minimal clarification,” “satisfactory requiring moderate clarification,” or “unsatisfactory requiring substantial clarification.”

Results

Of the responses given by the chatbot, 45.5% of the time the expert panel rated the response as “excellent,” 38.2% of the time the rating was “satisfactory requiring minimal clarification,” and only 11.8% and 4.5% of the responses were rated as “satisfactory requiring moderate clarification” and “unsatisfactory,” respectively.

Conclusion

ChatGPT provides reliable information to patients undergoing shoulder arthroplasty. As a complementary educational resource, AI shows potential by delivering accurate, evidence-based answers with little need for clarification and minimal bias. Despite limitations, the study underscores the promising role of AI-based platforms in healthcare.