ChatGPT can provide satisfactory answers to patient questions regarding biceps tenodesis

Abstract

Background

Shoulder pain often arises from pathology affecting the long head of the biceps brachii tendon, with biceps tenodesis recognized as a widely-accepted treatment. Research indicates a significant correlation between preoperative expectations and postoperative satisfaction, emphasizing the pivotal role of patient education. Although advancements in artificial intelligence (AI) technology, such as ChatGPT, hold promise for providing comprehensive health-related information, the accuracy can vary. This study aims to assess the quality and accuracy of ChatGPT's responses to patient inquiries about biceps tenodesis, offering insight on AI’s contribution to enhancing patients' understanding and expectation management.

Methods

A list of the 10 most frequently asked patient questions regarding biceps tenodesis were identified from the websites of various orthopedic institutions. Each question was individually entered into the most current version of ChatGPT at the time of data collection (v3.5), and the responses were recorded. The responses were then reviewed by four board-certified orthopedic surgeons and graded based on two evidence-based rating systems: the ChatGPT response rating system and the AI response metric. Additionally, the reading level required to fully comprehend the responses was calculated using the Flesch-Kincaid Grade Level assessment.

Results

Of the 10 responses evaluated, 1 was deemed excellent, requiring no clarification, and 7 were satisfactory, needing minimal clarification according to the ChatGPT response rating system. Two out of 10 responses were rated by surgeons as excellent, clear, comprehensive, and aligned with current literature. Seven responses were deemed good, being mostly clear and complete and in line with current literature. On average, a Flesch-Kincaid score indicated that 18.5 years of education were required to fully comprehend the responses, corresponding to the level of a college graduate.

Discussion/Conclusion

Most of the ChatGPT responses can provide satisfactory but limited information to answer patients' questions about biceps tenodesis. The reading level required to comprehend the response was too advanced for the average patient, leading to potential misunderstanding and misinterpretation of the response. Recognizing the limitations of ChatGPT to accurately and comprehensively answer patient questions can help guide surgeons when approaching discussions with patients regarding biceps tenodesis.