Multimodal fusion architectures for Alzheimer’s disease diagnosis: An experimental study

Abstract

Objective:

In the attempt of early diagnosis of Alzheimer’s Disease, varying forms of medical records of multiple modalities are gathered to seize the interaction of multiple factors. However, the heterogeneity of multimodal data brings a challenge. Hence, the role of artificial intelligence comes into play to provide the medical practitioner assistance in making diagnosis and prognosis. In order to be adopted as a clinical decision support system, interpretable or explainable model is important for healthcare professionals to trust the results. This study assessed various popular machine learning models under two multimodal fusion architectures to find the best combination in terms of both predictive performance and interpretability.

Methods:

Two architectures, early and late, also known as feature- and decision-level fusion were chosen for multinomial classification task. On top of the commonly used simple concatenation, this study employed weighted and hybrid weighted concatenation to fuse features within and across modalities under the two fusion structures. To test the efficacy of each model pipeline, the assessment was done according to their distinct foundations on which the models were built and each of their advantages was recognized. Classification metrics were unified and visualized into a pentagon to compare the overall performance of each pipeline. In addition, interpretability analysis was provided to quantify the importance of each modality and feature recognized by each model.

Results:

The potential characteristics of each type of pipelines in terms of prediction accuracy and ability to capture the relevant markers of each cognitive state were uncovered. In this particular healthcare application, the tree-based and linear models were the top 2 choices. Coupled with early and late fusion structure with weighted concatenation, reaching the balanced accuracy of 0.920 and 0.912, consecutively. The top 5 most important features revealed belong to Cognitive Test Scores and Neuropsychological Battery of Test modalities.

Conclusion:

This work contributes as medical applications of artificial intelligence evaluation to aid practitioners in understanding the capability of different fusion architectures with different classifiers in getting to know the use of machine learning in clinical setting. With accurate classification, early detection of Mild Cognitive Impairment and Alzheimer’s Disease can be achieved.