Readability, CPT Billing Accuracy and Diagnostic Coding Performance of ChatGPT Models in Hand Surgery.

Abstract

Background: Large language models (LLMs) like ChatGPT show their potential use in healthcare. However, limited research has been conducted on how effective these models are in hand surgery. Therefore, the purpose of this study is to compare ChatGPT-4, ChatGPT-4o and ChatGPT-o1 across patient education, billing and diagnostic coding in hand surgery. Methods: LLMs were tested using standardised prompts for five common hand conditions: distal radius fractures, trigger finger, carpal tunnel syndrome, cubital tunnel syndrome and ganglion cysts. Three tasks were evaluated: (1) patient education (15 frequently asked questions), (2) CPT code prediction and recognition of a supplied correct CPT code from de-identified operative notes, and (3) ICD-10 code prediction from clinical vignettes (HPI, physical exam and imaging). Readability was assessed using the SMOG Index, Flesch-Kincaid Grade Level and Flesch Reading Ease. Accuracy for billing and diagnostic tasks was measured using F1 scores. Repeated-measures and group-level statistical analyses were performed using Friedman, Wilcoxon signed-rank, Kruskal-Wallis and Dunn tests with Holm-corrected post hoc comparisons, as appropriate. Results: ChatGPT-o1 generated significantly more complex responses than ChatGPT-4 and ChatGPT-4o (adjusted p < 0.001), while responses to surgical questions were significantly more difficult to read than diagnostic questions (SMOG: adjusted p = 0.001; FRE: adjusted p = 0.024). All models achieved near-perfect accuracy in detecting CPT codes. At the three-character ICD category level, ChatGPT-o1 outperformed ChatGPT-4o (adjusted p = 0.001). At the four-character ICD subcategory level, ChatGPT-o1 outperformed both ChatGPT-4 and ChatGPT-4o (adjusted p < 0.001 for both). For exact ICD-10 code prediction, ChatGPT-4o and ChatGPT-o1 both outperformed ChatGPT-4 (adjusted p < 0.001), while ChatGPT-4o and ChatGPT-o1 did not significantly differ (adjusted p = 0.995). Diagnostic accuracy improved with the addition of physical examination and imaging information. Conclusions: Even though LLMs struggled with predicting billing codes and exact ICD-10 codes, they performed better at recognizing supplied correct billing codes and identifying broader diagnostic groupings. All models produced educational content above the recommended reading level. These findings highlight both the promise and limitations of LLMs in hand surgery. Level of Evidence: Level V (Diagnostic).