Evaluation of Online AI-Generated Foot and Ankle Surgery Information

Abstract

As a natural progression from educational pamphlets to the worldwide web, and now artificial intelligence (AI), OpenAI chatbots provide a simple way of obtaining pathology-specific patient information, however, little is known concerning the readability and quality of foot and ankle surgery information. This investigation compares such information using the commercially available OpenAI ChatGPT Chatbot and FootCareMD®. A list of common foot and ankle pathologies from FootCareMD® were queried and compared with similar results using ChatGPT. From both resources, the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL) scores were calculated for each condition. Qualitative analysis of each query was performed using the JAMA Benchmark Criteria Score and the DISCERN Score.The overall ChatGPT and FootCareMD® FRES scores were 31.12 ± 7.86 and 55.18 ± 7.27, respectively (p < .0001). The overall ChatGPT and FootCareMD® FKGL scores were 13.79 ± 1.22 and 9.60 ± 1.24 respectively (p < .0001), except for the pilon fracture FKGL scores (p = .09). The average JAMA Benchmark for all information obtained through ChatGPT and FootCareMD® were 0 ± 0 and 1.95 ± 0.15 (p < .001), respectively. The DISCERN Score for all information obtained through ChatGPT and FootCareMD® were 52.53 ± 5.39 and 66.93 ± 4.57 (p < .001), respectively. AI-assisted queries concerning common foot and ankle pathologies are written at a higher grade level and with less reliability and accuracy compared to similar information available on FootCareMD®. With the ease of use and increase in AI technology, consideration should be given to the nature and quality of information being shared with respect to the diagnosis and treatment of foot and ankle conditions.