Real-World Generalizability of Alzheimer's Volumetric MRI Machine-Learning Models: External Validation with British Data.

Abstract

Purpose: Assessing generalizability and performance of machine learning models in clinical settings is crucial. In this study, we aimed to test our models' performance on an external validation real-world clinical dataset and to evaluate the impact of magnetic field strength and brain volume normalization on the classification.

Methods: We validated two previously published models trained on public datasets (Alzheimer's disease [AD], mild cognitive impairment [MCI] and cognitively normal [CN] subjects) on a real-world clinical dataset from UK memory clinics (SLaM-BRC: 255 non-AD [subjects without cognitive complaints], 281 MCI and 711 AD).

Results: Our 'CN vs. AD' model showed similar performance when tested at different magnetic fields (1.5T vs. 3.0T: 87.1% of 93 subjects had the same class assignment). The volume-normalized 'CN vs. AD' model (87.7% balanced accuracy [BAC]) led to decreased performance in the SLaM-BRC (81.5% BAC) due to misclassifications of non-AD subjects with evidence of hippocampal atrophy. The non-normalized 'CN vs. MCI vs. AD' model's performance (initially 55.3% BAC) remained similar in the SLaM-BRC cohort (BAC: SLaM-BRC = 54.6%).

Conclusion: Volumes normalized to the estimated total intracranial volume led to a smaller difference in performance between internal and external datasets than non-normalized volumes. Our 'CN vs. MCI vs. AD' model performance remained the same, denoting robustness. These findings suggest that dataset and disease/diagnostic heterogeneities, magnetic field, and brain volume normalization may affect models' performance.