Chatbots in Limb Lengthening and Reconstruction Surgery: How Accurate Are the Responses?

Abstract

Background: Artificial intelligence-based language model chatbots are being increasingly used as a quick reference for healthcare related information. In pediatric orthopaedics, studies have shown that a significant percentage of parents use online search engines to find out more about the health condition of their children. Several studies have investigated the accuracy of the responses generated from these chatbots. The accuracy of responses with these programs in limb lengthening and reconstruction surgery has not previously been determined. Our goal was to assess the response accuracy of 3 different chatbots (ChatGPT, Google Bard, and Microsoft Copilot) to questions related to limb reconstruction surgery.

Methods: A list of 23 common questions related to limb reconstruction surgery was generated and posed to the 3 chatbots on 3 separate occasions. Responses were randomized and platform-blinded before rating by 3 orthopaedic surgeons. The 4-point rating system reported by Mika et al was used to grade all responses.

Results: We found that ChatGPT had the best response accuracy score of all 3 chatbots while Microsoft Copilot had the worst score, and this finding was consistent among all 3 raters.

Conclusions: Using the Response Accuracy Score, the responses from ChatGPT were determined to be satisfactory, requiring minimal clarification, while responses from Microsoft Copilot required moderate clarification.

Level of evidence: Level IV-diagnostic.