Expert evaluation of ChatGPT-generated responses to patient questions in conventional and fast-track total knee arthroplasty

Abstract

Background

Fast-track total knee arthroplasty (TKA) asks patients to understand early mobilization, early discharge, opioid-sparing analgesia, and selective use of drains and catheters. We evaluated expert-rated quality of ChatGPT responses to patient questions about these pathway-specific concepts alongside conventional TKA questions.

Materials and methods

In this cross-sectional evaluation, three arthroplasty surgeons independently scored responses to 50 conventional TKA and 20 investigator-developed fast-track questions in four domains on 5-point scales. The conventional questions were selected by a multidisciplinary panel; the fast-track questions did not undergo the same selection process. Response-level means were compared descriptively and with exploratory Mann-Whitney U tests.

Results

Seventy responses received 210 expert assessments. Recalculated mean ± SD scores across individual ratings were 4.99 ± 0.15 for accuracy, 4.98 ± 0.14 for completeness, 4.98 ± 0.17 for safety, and 5.00 ± 0.07 for readability. Each of the 20 fast-track responses received a score of 5 from each of the three raters in all four domains (Supplementary Table S1). Nevertheless, question-set construction differed and scores clustered at the ceiling, so the exploratory comparison does not establish equivalence.

Conclusions

Experts gave high ratings to the evaluated responses, including those on fast-track concepts. The observed data do not establish equivalence between question sets: the fast-track ratings require source-data verification, the sets were selected differently, and scores were concentrated at the maximum. Clinician review remains appropriate for protocol-dependent and safety-related advice.

Trial registration

Not applicable.