Multi-Modal Transformer Framework for Automated Knee Osteoarthritis Severity Grading From Radiographs

Abstract

To develop and validate a multi-modal, artificial intelligence (AI)-driven framework for automated grading of knee osteoarthritis severity from radiographs, integrating advanced detection, deep learning, radiomics, and clinical data. A total of 2290 patients from five medical centers were included, split into an internal cohort (n = 1549) and an external test set (n = 741). Knee radiographs were preprocessed with histogram normalization, contrast-limited adaptive histogram equalization (CLAHE), and noise filtering. Four detection models, RT-DETR, SAM, DINOv2 + ViTDet, and Grounded-SAM/MedSAM, were evaluated for anatomical localization, with Grounded-SAM/MedSAM selected for optimal intersection over union (IoU) and mean average precision (mAP). Radiomic features (n = 215) were extracted from detected regions of interest (ROIs) using the Standardized Environment for Radiomics Analysis (SERA) platform, while deep features were derived from four advanced extractors, with MedSAM + feature aggregator selected for highest classification accuracy. Features underwent harmonization, variance filtering, and selection via least absolute shrinkage and selection operator (LASSO), principal component analysis (PCA), or mutual information. Clinical, radiomic, and deep features were fused and classified using CoAtNet, ConvNeXt V2, EfficientNetV2, RepLKNet, and Swin V2 Transformers. Model selection was performed within a nested cross-validation scheme on the internal cohort, and the external cohort was accessed only once after the final configuration had been fixed. Model performance was assessed by accuracy, F1-score, and area under the receiver operating characteristic curve (AUC), with interpretability provided by SHapley Additive exPlanations (SHAP) analysis and a nomogram. Grounded-SAM/MedSAM achieved the highest detection performance (external IoU = 90.83% [95% confidence interval, CI, 89.41–92.15], mAP = 93.67% [95% CI 92.38–94.82]). MedSAM deep features outperformed others (external F1 = 82.37% [95% CI 80.56–84.18], AUC = 0.902 [95% CI 0.888–0.916]). Fused features with Swin V2 + LASSO yielded the best results (external accuracy = 94.6% [95% CI 93.5–95.7], F1 = 94.4% [95% CI 93.3–95.5], AUC = 97.2% [95% CI 96.4–98.0]), significantly surpassing single-modality models. SHAP analysis confirmed balanced contributions from deep, radiomic, and clinical features, while the nomogram achieved a C-index of 0.970 [95% CI 0.958–0.980] with excellent calibration. This framework demonstrates high accuracy, robustness, and interpretability in knee osteoarthritis severity grading from radiographs, offering strong potential for real-world clinical integration.