Visual-language artificial intelligence system for knee radiograph diagnosis and interpretation: a collaborative system with humans.

Background: Large language models (LLMs) have shown promising abilities in text-based clinical tasks but they do not inherently interpret medical images such as knee radiographs.

Purpose: To develop a human-artificial intelligence interactive diagnostic approach, named radiology generative pretrained transformer (RadGPT), aimed at assisting and synergizing with human users for the interpretation of knee radiological images.

Materials and methods: A total of 22 512 knee roentgen ray images and reports were retrieved from Massachusetts General Hospital; 80% of these were used for model training and 10% were used for model testing and validation, respectively. Fifteen diagnostic imaging features (eg, osteoarthritis, effusion, joint space narrowing, osteophyte) were selected to label images based on their high frequency and clinical relevance in the retrieved official reports. Area under the curve scores were calculated for each feature to assess the diagnostic performance. To evaluate the quality of the generated medical text, historical clinical reports were used as the reference text. Several metrics for text generation tasks are applied, including BiLingual Evaluation Understudy, Recall-Oriented Understudy for Gisting Evaluation, Metric for Evaluation of Translation with Explicit Ordering, and Semantic Propositional Image Caption Evaluation.

Results: RadGPT, in collaboration with human users, achieved area under the curve scores ranging from 0.76 for osteonecrosis to 0.91 for arthroplasty across 15 diagnostic categories for knee conditions. Compared with the baseline LLM method, RadGPT achieved higher scores, specifically 0.18 in BiLingual Evaluation Understudy score, 0.30 in Recall-Oriented Understudy for Gisting Evaluation-L, 0.10 in Metric for Evaluation of Translation with Explicit Ordering, and 0.15 in Semantic Propositional Image Caption Evaluation, which is significantly higher than the baseline LLM method, demonstrating good linguistic overlap and clinical consistency with the reference reports.

Conclusion: RadGPT has achieved advanced results in knee roentgen ray image feature recognition, illustrating the potential of LLMs in medical image interpretation. The study establishes a training protocol for developing artificial intelligence-assisted tools specifically focusing on the diagnosis and interpretation of knee radiological images.