Refining patient education on autologous chondrocyte implantation for chondral lesions of the knee: A fine-tuned ChatGPT-4o model improves readability and quality

Purpose

Autologous chondrocyte implantation (ACI) is a complex procedure for cartilage defects, requiring patient understanding of treatment and recovery, as health literacy impacts outcomes. This study evaluated the quality and readability of AI-generated ACI materials using ChatGPT-4o as adjuncts to physician-led education. We compared responses from the native model and a fine-tuned version and hypothesised that the fine-tuned model would provide improved quality and readability.

Methods

Twenty-two frequently asked questions were identified using Google's ‘People Also Asked’ feature. Two ChatGPT-4o configurations were evaluated: the native model and a fine-tuned version (ACI guide) optimised by instruction-based fine-tuning and reinforcement learning from human feedback. Two orthopaedic surgeons independently scored the responses. Quality was assessed using the DISCERN criteria and readability by the Flesch Reading Ease Score (FRES) and Flesch-Kincaid Grade Level (FKGL). Interrater reliability was determined using intraclass correlation coefficient (ICC) in a two-way mixed-effects model.

Results

The fine-tuned ACI Guide outperformed the native ChatGPT-4o on all parameters. The native model produced poor-quality responses with a mean DISCERN score of 35.29 ± 5.0 (range: 23–45), while the ACI Guide achieved a significantly higher score of 43.18  ± 3.92 (range: 34–53; p < 0.001), reflecting moderate quality. Regarding readability, the native model reached FKGL of 13.45 ± 1.30 (university sophomore level). In contrast, the ACI Guide achieved FKGL of 9.25 ± 1.64 (9th-grade level). The FRES was also significantly higher for the ACI Guide (49.59 ± 10.44) than the native model (35.68 ± 5.08; p < 0.001). Interrater reliability was strong (ICC = 0.767), indicating good agreement.

Conclusions

ChatGPT-4o's responses were of poor quality and written at a readability level substantially exceeding recommended thresholds for patient education materials, limiting their applicability in clinical communication and patient education. Fine-tuning ChatGPT-4o improved the readability and quality of ACI patient education materials, generating content closer to the 8th–9th-grade level. It may serve as a useful adjunct to physician-led education in enhancing patient understanding of complex orthopaedic procedures.

Level of Evidence

Level V.